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Introduction 



This document reports the proceedings of a conference on assessing high order thinking 
skills organized by Northwest Regional Educational Laboratory and cosponsored by: 

Northwest Regional Educational Laboratory (NWREL) 
Oregon Department of Education 

Office of the Superintendent of Public Instruction, Washington 
Northwest Evaluation Association (NWEA) 
ESD 112 in Vancouver, Washington 
Idaho Association of School Administrators 



The conference was held October 1 and 2, 1987 in Clackamas Oregon. The program for 
the conference is reproduced in Appendix A. The list of participants is provided in Appenoix 
B. The purpose of the conference is best summed up by Gary Estes, Director of Evaluation & 
Assessment at NWREL, program introduction. 

The Northwest Regional Educational Laboratory, in conjunction with five 
regional associations and departments of education, is pleased to welcome you to 
two days of presentations and workshops on assessing higher order thinking skills 
(HOTS). Good assessment is essential if we are going to make decisions about 
the instructional needs and progress of students. One goal of this conference is 
to assist people who are deciding how to assess this critical area in our schools to 
make more informed decisions and to produce higher quality assessments. 
Another goal of this conference is to bring together people interested in the 
assessment aspects of thinking skills. One outcome might be a cooperative effort 
in item banking. 

The conference has been divided into two parts: Issues and Practices. The 
afternoon of the first day is designed to elicit perspectives about assessing higher 
order thinking skills. Issues such as how should KOTS be assessed, the quality of 
existing tests, how do we define HOTS and what will the future bring will be 
discussed. This part will set the stage for the second day's activities. 

The second day emphasizes practices. There are three strands: daily, less formal 
classroom assessment; formal assessment procedures that may be used at the 
district level; and training on how to write test questions which assess HOTS. 

The presenters and participants in the eighteen sessions will make each of the 
sessions a worthwhile exploration of the issues and practices for assessing higher 
order thinking skills. 

Once again, welcome! We are looking forward to a productive conference 
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These proceedings were produced by persons assigned to each session to take notes on 
the essential points covered by each session. All presenter handouts are included. 

One outcome of this conference might be a cooperative NWEA-type project in the area 
of higher order thinking skills. NWEA is a consortium of school districts which work together 
to solve problems of mutual interest. Projects in the past have been in the areas of basic skills 
(resulting in good-quality item banks in the areas of reading, math and language arts) and 
science (which currently has addressed curriculum issues and has produced 7,009 test questions). 
The exaci nature of the cooperative effort is developed by those districts, state departments of 
education and other organizations which participate. We invite readers who may be interested 
in such an effort to contact NWREL (503-275-9500) or NWEA (206-839-3932). 



9i 



Assessing Higher Order Thinking Skills — Issues and Practices 

Keynote Address 
Stuart Rankin 



Dr. Rankin's objective for his address was to clarify assessnaent issues surrounding 
measuring higher order thinking skills (HOTS). He did not address the need to teach HOTS 
because he assumed that the audience recognized the need for this. Given that we need to teach 
HOTS, then it follows that we must assess students so that we can plan the proper instruction. 

Dr. Rankin asserted that most of the major issues surrounding HOTS assessment are 
issues of validity how do we obtain a valid view of how students think? Dr. Rankin used 
his "Rankin-Hughes Framework Of Thinking Skills" to illustrate the following assessment issues 
surrounding HOTS: 



1. The content to be co\ered by the assessment should be based on some theoretical 
framework. 

We should have a rationale for what we teach and what we assess. This rationale is 
best established by using a good framework of HOTS skills. Dr. Rankin briefly discussed 
Bloom*s taxonomy and the framework surrounding the Philosophy For Children Program. Then 
he used his own framework as an illustration of a reasoned and comprehensive framework. 
(The attached paper entitled "The Rankin-Hughes Framework of Thinking Skills" discusses this 
framework in more detail.) 

2. We should consider the fuli-range of HOTS skills in our instruction and assessment, 
not just the easiest ones to teach or assess. 

Dr. Rankin asserted that many current tests (especially those in paper and pencil 
format) test a narrow range of HOTS skills, namel>, those that are amenable to assessment in 
that format. He once again used his framework to illustrate that thinking skills consist of man> 
more areas than logical syllogisms, identifying assumptions and making inferences. 

For example, we should not ignore such things as thinking processes such as concept 
formation, oral communication, problem solving and decision making, metacognition, creativity, 
critical thinking; and the affective side of HOTS. 

3. We should be very careful svhat we assess because assessment can drive the 
curriculum. 

Dr. Rankin sees thinking as a very complex task. We have a pool of skills, abilities, 
dispositions and processes, to be drawn from as needed. We should not view HOTS as being a 
fixed set of skills that are applied the same way for each use. He has a holistic view of how we 
solve problems we draw what we need as we need it. Skills should not be taught or assessed 
in isolation except to sharpen them. Therefore we should try to assess the thinking process as a 
whole rather than as separate skills. If we view thinking as separate skills, and worse yet, 
separate skills as listed by some fixed taxonomj, then we threaten to drive the curriculum in 
that direction. 
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A related issue is that any assessment is a sample of skills from a domain. We cannot 
assess the entire domain. The curriculum becomes restricted if Nve treat the content of the test 
as the entire domain of interest. 

4« The purpose of assessment should be to pla^ instruction not to screen students. 

Dr. Rankin feels that using tests of HOTS to scieen students is nonproductive. If we 
use assessment devices at all, it should be to assist teachers to know how to help students. 

5« We should spend more time developing test specifications* 

Test specifications provide a blueprint for a test what is to be covered, how each 
skill will be assessed, and how may questions or exercises will be written to measure each skill. 
Dr. Rankin asserts that test specifications, in addition to helping develop a good test, can zho 
be useful instructionally. Test specifications should clearly define each skill, should provide a 
sample question or exercise that illustrates how to measure that skill, criteria for judging 
whether the student has demonstrated the skill to be measured, and what the iraplicaiions for 
instruction are if the student does not demonstrate the skill. In addition, these test 
specifications should cover all areas of importance not just those easy to measure. 

In order to develop good test specifications (and in the process, good instructional aids) there 
needs to be consultation between curriculum people and test developers. The curriculum people 
need to specify whal should be measured (and taught). Then the groups can work together to 
figure out how to measure it adequately. 

6. We should try to control for prior knowledge when development assessments of 
HOTS. 

The issue here is thai student knowledge can alter a task so that instead of measuring 
thinking, it really measures something else such as recall of knov;ledge or vocabulary. Dr. 
Rankin pointed out that kids with good vocabularies do well on thinking skills tests. 
Vocabulary level can contaminate the results. 

7. We need to consider assessing skills in more formats than multiple-choice. 

If we really want to gather information about most of the more interesting thinking 
skills, we need to consider other assessment formats than multiple-choice. Other formats 
include short answer, essay and performance tasks. 

8. Our notions of what thinking skills are should not be fixed. 

We are in a continual process of better defining what thinking is and what thinking 
skills are. As we reconceptualize this area we should be redefining our taxonomies, teaching 
strategies and assessment activities. If our skills lists become too fixed, we are threatened by 
rote approaches. We need to be continually thinking about thinking. We may be better able to 
define thinking by developing mathematical models of the thinking process. Dr. Rankin invited 
the audience to pursue this direction. 

9. We should disaggregate information about the HOTS perfcrm^*Qce of students. 

Dr. Rankin feels that we should use the information we coliect about students to 
improve instruction. One way to do this is to look at the relative performance of various 
groups of students. We should confront head-on issues such as difference in performance 
between ethnic groups. 



I0« Test developers should be critical tliinlcers too* 



We need to be constantly thinking about what we are doing and why. We need to 
consider alternatives, be creative and do what makes sense. 



Panel Response 

Dr. Stephen Norris, Dr. Edys Quellmalz and Dr. Ken Bumgarner responded to Dr. 
Rankings comments. 

Dn Norris indicated that he agreed with Dr. Rankin on the following points: 



1. Evaluation must be based upon a comprehensive theoretical framework of good 
thinking. This framework must have the flexibility to evolve. 

2. We need to conceive of good thinking as selecting from a pool of skills those which 
are appropriate for a problem rather than as rigidl> applying a sequenced set of skills. 

3. We need to focus our attention on contexts of reasoning and on the complexity of 
coordinating many skills in the solution of problems in content, rather than on the 
use of decontextualized, isolated skills. 

4. More attention needs to be paid to the attitudes and dispositions associated with good 
thinking. 

5. We must remember that critical and creative thinking do not mark off compleieh 
separate acts. They overlap and are mutually supportive. 

6. Be careful that your thinking tests do not merel> test for background knowledge. 

7. Teachers and test developers need to be good thinkers themselves. 



Dr. Norris emphasized that choice of theoretical frameworks is ver> important, some are 
better than others. In particular, use of Bloom's Taxonomy to conceptualize good thinking (a 
use for which Bloom never intended it) can lead to serious problems: 



1. By "knowledge** Bloom means mere recall. In its everyday use and use in much 
educational theorizing, knowledge means far more than this. 

2. Comprehension is near the bottom of Bloom's taxonomj, but comprehension can be 
one of the most complex tasks in which human beings engage. 

3. The categories of the taxonon^y are too vague to provide much guidance for testing, 
for example, the Evaluation category provides no criteria for making evaluations, 
students' knowledge of which should be tested. 

4. In general, the noJon "Higher Level Thinking" is dangerous. Many so-called low 
level skills (e.g., reading and observation) are very complex when done properh. 
Much depends on the context* 



Finally, Dr. Norris commented that whereas Dr. Rankin had claimed that critical 
thinking is not a bunch of skills, it is an attitude, critical thinking is best thought of as a 
combination of both skills and attitudes. 



Dr. Quellmalz expanded on several of Dr. Rankin's points. With respect to theoretical 
frameworks, she feels tb^t there are currently four to five good, comprehensive theoreUcal 
frameworks. We need criteria for evaluating them. We also need to look to other fields, such 
as cognitive psychology to refine frameworks. Second, Dr. Quellmalz emphasized the need to 
assess thinking skills vithin some meaningful context purposeful, sustained efforts. Third, 
she beiieveb that some skills are general and can be applied across content domains. 

Fourth, she agreed with Dr. Rankm that we need to broaden our assessment procedures 
to include other formats than multiple-choice. She suggested portfolios of student work> 
interviews, and learning logs. 

Dr. Bumgarner agreed with Dr. Rankin that we should use assessment for instructional 
planning, but, it also promotes what is of value. Therefore, it is important to assess it so that it 
is taught. Dr. Bumgarner feels that we should stress all thinking skills not just "higher order" 
ones. Recall of knowledge is important too. He also believes that there is an increasing gap 
between the general state of information on teaching and assessing HOTS and common 
classroom practice. 

Dr. Bumgarner added to the list of issues presented by Dr. Rankin. We need to consider 
the effect of testing on students, both In terms of self-concept and in terms of what the types 
of tests we give to students tell students about what is important. An example of the latter 
point is that if we give students tests having only one right answer, they ma> come to believe 
that everything has a right answer, an unrealistic expectation of the real world. 

Finally Dr. Bumgarner stressed that we are currently too dependent on commercially 
de\ eloped tests. This, as Dr. Rankin pointed out, leads to reductionism. We need to continue 
to develop in teachers the ability to be valid assessors because the on-going interaction of 
teachers and students should be the arena for continual assessment and instructional planning. 
Finally we need to establish public faith in teachers' ability to assess learning. 
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THE RANKIN ^ HUGHES FRAMEWORK* OF THINKING SKILLS 



The major thinking processes have much in common. Problem solving, 
decision making, disciplined inquiry, concept formation, principle formation, 
comprehending, and composing generally begin with an unresolved problem, 
disequilibrium, or indeterminate situation and move to a resolution, new 
^ equilibrium, or new meaning. 

In one sense the common features of the processes can be derivec^ by 
examining the steps in each of the major processes and noting their similari- 
ties*. In another sense the common process is generic to, imbedded in, and 
Indeed drives the major thinking processes. 

m 

Rankin and Hughes have designed a framework which spells out steps or 
stages that appear to be common to all of the major thinking processes. The 
framework also shows a way the individual thinking skills can be related to 
the processes. Some skills are more useful at certain steps \n a process 
than at others. The framework, however, is not intended to restrict the 
^ place in the process where a skill may be used or to Imply that each process 

always follows the same ordered sequence of skills. 

This framework does help to reinforce the idea that thinking skills are 
part of a larger process and that, although they can be described and some- 
times taught in isolation, they are best viewed as tools to be sharpened and 
9 used in constructing meaning, solving problems, and creating products and 

ideas. 

It should be noted that thinking skills are often imbedded in each 

other. For example, comodring and classifying are skills that may be used as 

part of -^icognizing patterns and relationships or of inferring. Thinking 

® skills are alike in that they usually involve mental acts which combine two 

or more representations of elements. The elements represented may be 

objects, units of experience, or observations; they may be attributes^ 

qualities or characteristics; they may be symbols, concepts, or principles; 

they may be simple or complex; an^ they may be single items or patterns or 

^ relationships. 
9 

The generic process presented in this framework has seven steps or 
stages/ They are (1) focus, (2) gather information, (3) organize information, 
(4) analyze information, (5) generate ideas, (6) synthesize, (7) evaluate 
and apply. Some thinking processes will involve all of the stages, while 
^ other thinking processes only involve some of the stepv depending on how 

complex the goal attainment becomes. 



♦Designed by Stuart C. RanK^n, Deputy Superintendent, Detroit Public 
Schools, and Carolyn S. n^gnes, Assistant Superintendent, Oklahoma City 
Pi;blic Schools for the Association for Supervision and Curriculum 
Development, 1986. 



Rir 



Outline of Framework 



Focus ^ 

- Sense problem 

- Define problem 

- Set goal 

Gather Information 

- observe 

- Recall 

- Question 

Organize Information 

- Represent 

- Compare 

- Classify 

- Order 

Analyze Information 

• Distinguish and clarify components and attributes 

- Determine accuracy and adequacy of arguments 

• Recognize patterns and relationships 

- Identify central element 

Generate Ideas 

- Infer 

- Anticipate 

- Discover relevant outside structures 

- Restructure 

Synthesize 

- Summarize 

- Integrate 

- Develop outcome 

Evaluate and Apply 

- Establish criteria and standards 

- Verify 

- Revise 

- Transfer 
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I. Focus 
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The first step in a thinking process is focusing attention. 
When ^\\e mature thinker senses a problem, he or she will often define the 
probUin and set a goal before proceeding to gather or to process information. 
When the problem Is less complex, or when the thinker is less mature, problem 
sensing and defining may be unconscious steps which lead the thinker immediately 
to focus on a goal and begin to gather information. 

Sense Problem 

The skill of problem sensing is the ability to discern a lack of fit 
between what Is and what is needed; a growing awareness of discrepancy, 
diset^unibrlum or need; or a consciousness that something is niissing or 
needed to restore or to create a meaningful whole. This jikill involves 
recognition of a need to know or to understand. It may include awareness of 
a desire to produce or to experience something. Developing the skill of 
problem sensing is based on attitudinal components of curiosity or concern 
in combination with knowing the factors critical to a particular situation. 
The situation for problem sensing may be as simple as awareness of a fuel 
Indicator approaching "empty" or as complex as recognising when a 
politician's words and actions are incongruent. 

Define Problem 

As an individual moves from consciousness of a need, problem, or 
discrepancy to engagement with it, he or she focuses attention on the 
situation. Direct instruction and practice can result in growing ability to 
focus and control attention. Identification of examples or relevant 
variables helps to bring a problem into focus. Such focus enables the 
thinker to determine whether a prow.em does indeed exist, and to confront the 
problem and to perceive it dearly. Developing skill in defining a problem 
includes determining assumptions and making an initial estimate of success 
criteria so that one will know when the problem or discrepancy is resolved. 
Problem definition may be stimulated by such a question as "What is it that 
leads you to wonder whether you can trust what that politician says?" 

Set 6oal 

Once an individual has b€COff»e conscious of a need, problem or discre- 
pancy and confronted various aspects of the situation, a goal is needed to 
guide and justify effort. This is a commitment of energy which involves 
determining direction and identifying possible outcomes. Goal setting 
involves anticipating types of information and resources needed, identifying 
processes to use, and choosing a tentative medium of expression* As stu- 
dents grow in metacogniti ve awareness, goal setting will include planning a 
thinking strategy: the combination of thinking skills needed to achieve the 
purpose. Determining direction may be guided by a question such as, "How 
could we determine whether that politician's actions are consistent with 
what'he says in this speeches?- 
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2. Gather ipfpr.T.at icr 



The foundation skills for thinking are those which make available to 
the mind what one thinks about. Information gathering skills provide the 
basic substance or content which becomes the raw material for cognitive pro 
cessing. The skills which make that substance available to the thinker are 
observing to gather first-hand sensory input, recalling to activate prior 
knowledge, and questioning to seek out new information. 



Observe 

The ability to focus attention and perception is a prima'-y element in 
the skill of observing — the use of the senses to gain direct, first-hand 
Information. Sustained focus and systematic search add power to the skill 
of observing. When words are used to describe and- differentiate sensory 
experiences, awareness grows helping students become better observers and 
better reporters of their observations. When observing, the ability to nota 
unfamiliar characteristics or terms can provide valuable clues for further 
thinking. The skill of observing is guided by^asking such questions as, 
"What do you notice about ?" 



Recall 

Recalling is the skill of accessing or activating prior knowledge from 
memory. It may be an active, thoughtful process of reconstruction — 
systematically recounting soecific information or events from prior 
experience, or it may be simple association such as recalling a name or 
number. The content for recalling may be experiences, information, con- 
cepts, structures or categories. 



Question 

While observing and recalling provide information which is immediately 
available to the senses or within one's previous knowledge or experience, 
questioning is a primary skill for gathering new information. Questioning 
is guided by awareness of the need for relevant, accurate, specific infor- 
mation. The clearer one's understanding of a problem or issue, the more 
focused will be the questions one uses to elicit information. Skllltui 
questioning, based on what one needs to know, enables one to gam optimum 
value from reading, listening, interviewing, or discussion. 



3. Organize Information 

The limited number of isolated bits of information which one can keep 
in mind at a t: .ie leads to the need to organize information. Another reason 
for organizing information is to develop new insights and understandings. 
Representing is an organumq skill which enhances ones ibility to com- 
municate and to retrieve information when it is needed. The organizing 
skills of comparing, classifying, and ordering build a foundation for infor- 
mat ion analysis. 

" 15 
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Represent 

An early step 1n processing information which has been gathered is 
translating it into some form which is useful or appropriate to the task. 
Representing is changing the form but not the substance or value of the 
information. A name or label may be assigned to represent it. Outlines, 
graphs, tables, drawings, models or mental pictures may be used to represent 
the information. The thinker may represent information using logical, 
mathematical, musical or visual symbols. Intentional mental encoding of infor- 
mation so that it can be easily retrieved may include multi-sensory recall 
activation also known as deep processing or the use of mginory frameworks 
such as pegwords, familiar place clues, number/word clues, or other mnemonic 
devices. 

Compare 

One compares when one identifies likenesses of similarities. 
Contrasting is identifying differences. Developing proficiency in comparing 
and contrasting is enhanced by teaching them separately. In this framework 
the skill of comparing has been expanded to include contrasting as well as 
comparing. This basic skill for organizing information forms the foundation 
for developing concepts and for classifying information as well as for the 
thinking skills needed to analyze, and to generate ideas. 

Classify 

Classifying is identifying examples of a concept by ensuring that all 
concept attributes are present. The skill of classifying requires the 
knowledge (or invention) of a definition or concept appropriate for the infor- 
mation at hand. A component skill for classifying is grouping. Grouping is 
putting together items which one perceives to be related by some common 
characteristic(s). Grouping is based on comparing and may lead to classifying 
if a concept label or name is given to the group. 

Order 

One orders by first establishing a scheme or criterion for sequencing 
information. The criterion may be based on such physical factors as position, 
size, intensity, or duration, or it may be based on abstract qualities such as 
preference, priority or agree"ient. Once the scheme or criterion is deter- 
mined, it is used to place information in a sequence. 



4 . Analyze Information 



Analysis requires a 'oo^ mside the components to distinguish among the"", 
clarify them, determine thetr adequacy, or rearrange them. It involves the 
recognition of relationships and the identification of central elements. 
Examination is made of parts and patterns in order to understand them and to 
suggest possible solutions. The skills of analyzing information are central 
to critical thinking. 
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Distinguish and Clarify Components and Attributes 

The functions of different elements in a situation must be made clear. 
Distinctions can be made between facts and opinions, between inferences and 
assumptions, and among findings, conclusions, and implications. Attributes 
and meanings may require identification and clarification. 

Determine Accuracy and Adequacy of Arguments 

A fundamental step of analysis is establishing accuracy. Some elements 
may be biased, false, or ambiguous. Others may be inconsistent, irrelevant, 
or contradictory. Claims may be unwarranted. Omissons may be noted which 
limit comprehensiveness or break the chain of an argument. The accuracy of 
facts and the reliability of sources must be determined. 

Recognize Patterns and Relationships 

Patterns can be seen among elements whether those elements are concepts, 
objects, principles, attributes, qualities, or actions. Among the components 
various relationships m^y be identified which sre temporal or spatial, sequen- 
tial or correlational, or transitive or syllogistic. Casual or hierarchical 
relationships may be found. Analysis requires the recognition of these 
patterns. Comprehension and recall are enhanced when readers can recognize 
frequently used paragraph patterns such as comparison, classification, 
sequence, or causation. The study of number patterns and geometric patterns 
enhances the understanding of mathematics. 

Identify Central Element 

Many situations have a central theme or main idea. The essence of 
something may be described as Its kernal, or key element, or critical inci- 
dent. It may be a central fibre, a musical theme, a color or geometric pat- 
tern, of some other unifying element that adds meaning or impact to the who^e. 

5. Generate Ideas 

Where analysis looks mside a situation, generation of new ideas requires 
one to extend and expand, extension projects beyond the situation, and expan- 
sion makes distant connections that may lead to a whole recasting of the 
problem situation with additional components into a new structure. Extending 
depends upon inference and anticipation, while expanding makes use of inven- 
tion, discovery, imagination, and creativity. 

Infer 

New meanings, relationships, and characteristics can be implied by or 
derived from a given situation. Generalization, conclusions, inductions, 
and deductions lead to the new understandings. New meanings of a passage are 
inferred by extending the meanings contained in and supported by the passage, 
A casual relationship may be hypothesized about two events which are known to 
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have occurred but the relationship is not given; ofcvious experiences may 
suggest the relationship. Hypotheses about the characteristics o<- attributes 
of a component or subject may be supported by relating clues in the situation 
to previous knowledge, to experiences, or to the opinions of authorities. 

Anticipate 

Trends can often be projected. Possible results or occurrences can be 
identified. Probabilities may be determined for those possible consequences. 
Estimates of the magnitude or strength of a relationship can be made based on 
the situation and on oast experiences. Predictions and forecasts can be made 
beyond on estimates and probabilities. Though anticipating includes the skill 
of inferring, it projects further beyond the known and requires identification 
of necessary or sufficient conditions. 

Discover Relevant Outside Structures 

When a problem cannot be resolved by analysis or extension, or when a 
more creative solution is ouqht, it is necessary to look beyond the elements 
of the problem situation. One searches in past experiences and present obser- 
vations which are external to the problem situation in order to discover or 
invent a connection (isomorphism) with the problem situation. This type 
of thought usually occurs following withdrawal of the focus of attention on 
the problem situation. It may require the restraining of impulsivity or the 
withholding of judgment. Robert Penn Warren or.c«. called it, "Stop scratching 
in order to build up the itch." The lessening of- focus appears to let the 
mind be free to roam about to metaphorical situations taking advantage of 
imagination and creativity to find unexpected new suggestions that can lead to 
solutions. These new connections are isomorphic to some part of the problem 
solution in that there is a correspondence of some elements and some underlying 
structure. 

Restructure 

Further examination of the relevant outside structure in respect to the 
problem situation can produce ^naioo'es, metaphors, and other models which in 
turn may furnish hypotheses at)out possible solutions. This use of outside 
models serves to restructure or recast the indeterminate or unresolved problem 
situation or disequilibrium jnto i new structure which can form the basis of a 
solution. This restructur* nq mav be quite creative and is sometimes called a 
paradigm shift. Sometimes ti js'not just the elements of the problem situation 
that are recast, but the description or understanding of the problem itselt theC 
is restructured. 

6. S-,nthesize 



The results of ana'>/:tnc. 4nd generating ideas are new themes, 
meanings, structures, a^d design features which need to be pulled together to 
create a new understand\n9. product, or solution. Summaries are made of the 
key components; the coc'poneois ire integrated mto a meaningful design; and 
the actual outcome is developed. 
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Sunwarize 

The components, bindings, or design features which were discovered or 
created during analyses, and generating ideas are stated or represented in 
a condensed, concise, and comprehensive manner. The main metaphor for the 
poem Is identified; tKe key elements of the decision are determined; the parts 
of the solution are named; the colors and the materials are chosen. • 

Integrate 

To Integrate is U put the parts together in a whole. As does sum- 
marizing, this skill requires identification of the parts, but it also requires 
the organization of the parts so that their interrelationships and struc- ® 
tures form a meaningful whole. Complete integration goes beyond internal 
organization of the parts and requires that the new whole fits properly into 
its external world as well. The outline is written; the sketch is made, the 
clay model is formed; the dance movements are put in order. 

Develop Outcome 

When an integrated designed of a solution is formed, the synthesis is not 
finished until the full development of the outcome is accomplished. This 
skill varies with the medium and form of the outcome, but in each case it 
requires selection, execution, and completion of the details of the product. ^ 
All refinements of the meaning are made; the picture is painted; the decision 
is made; the concept is formed. 

7. Evaluate and Apply 

A full goal-oriented thinking process is not complete until the outcome 
is evaluated and applied. Standards are set for internal and external valid- 
ity and the product is verified with respect to these standards. Revisions^ 
are'made where necessary. When the criteria are met, the outcome is availabe 
for attaining the goal and for applying or transferring to other situations. 

Establish Criteria and Standards 

Two kinds of criteria must be set in order to judge the quality or ade- 
quacy of the solution, decision, meaning, or product. First, the outcome must 
have internal validity. It must be produced in the domain of the problem; it 
must resolve the disequilibrium that motivated the process; and it must be @ 
usable by the one who produced the outcome. Sometimes these standards for 
internal consistency are e«Dlicit, sometimes implicit. Second, the outcome 
must work in the real world. Standards for quality of an art work may be 
used The finding or meaning may reinforce or refute other knowledge. The 
solution may have wide or narro- applicability. It may be easy or hard to 
replicate. Criteria are used to set standards against which the outcome ^ 
is to be judged. Standards, to be most useful, should be measurable. 
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Verify 

After the standards are set, they are used to judge completeness, 
quality, and range of applicability. Outcomes are verified by comparing the"i 
with standards to determine if the solution, meaning, or product is consistent 
with the conditions accepted when the goal was set. Assumptions ana delimita- 
tions must be met. Also, the outcome must be of value, e.g., useful, beauti- 
ful, scarce, original. In addition it must resolve the initial problem 
situation. Both the internal and the external criteria must be met. 

Revise 

Evaluation results in decisions to accept, reject, or modify the whole 
outcome or its parts. If no revisions are needed, then the whole is accepted. 
Often some components (elements, structures, details) are retained, others are 
eliminated or replaced and still others are modified. Sometimes new components 
must be added. The determination of whether revis-ions are needed, of the 
reasons for the revisions, and of the areas where revisions are needed is an 
evaluation skill. The actual revision leads one back to earlier steps in the 
thinking process. Whether one returns to problem definition, goal setting, 
information gathering, analysis, extension, or expansion is determined by the 
evaluation decision to revise. 

Transfer 

When the outcome is finally accepted, that acceptance may be a new 
meaning which itself is the aoolication of the thinking process. Often out- 
comes can be used repeatedly for other similar situations. One assimilates or 
accommodates the outcome and it becomes part of experience. If stored well in 
memory, it will be retrievable for application to like problem situations and 
transferable to some situations quite unlike the problem situation. Such 
transfer requires (1) experiencing the process, (2) storing in a retrievable 
fashion, and (3) using thinking skills from the generating ideas category to 
help bridge the gaps. A final apoHcation of any solution or outcome may be 
the identification of a new problem. 
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Writing MuUiple-Choice Critical Thinking Items (part 1) 
Stephen P. Norris 



Dr. Norris suggested that while multiple choice tests of critical thinking have 
limitations, especially when used to provide information on specific information on 
specific individuals, they can when constructed properly provide valuable information 
groups. Multiple choice items do not, however, test for the orchestration of thinking 
skills on complex problems nor test for critical thinking dispositions. 

To frame the discussion about writing good critical thinking items, Dr. Norris 
distributed a handout. Some Guidelines for Writing Multiple-Choice Critical Thinking 
Test Items. The greatest part of this session followed the "Suggestions for Specific 
Aspects of Critical Thinking" section of the handout. Key points in this section were 
illustrated with examples from specific test items in published tests. 
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Some Guid el ines for Writing MultiDle-Choice 
Critical Thinking Test Itepis 



9 



from Stephen P. Norris 
and Robert H. Ennis, 
Evaiuating Critical Thinking . Pacific Grove, CA: 
MiLwest. (In Press) 

by 

Stephen P. Norris 
for 

Assessing Higher Order Thinking Skills: 
Issues and Practices 
The Northwest Regional Educational Laboratory 
October 1-2, 1987 



General Suggestions 

1. Start with a good conception of critical thinking. 

2. Identify {hose aspects of critical thinking that are most important in your situation. 
I Indicate relative importance with a weighting scheme. 

3. Decide on the purpose of the test (testing for mastery, comparing students wif'i 
others via norms, making placement decisions, diagnosis, pretest-posttest compari- 
sons). 

B 4. Decide whether you want to test for critical thinking in a particular school subject 

or in the context of general knowledge. 

5. Provide adequate context. 

6. Three or four alternative ai:swers are preferable to two because of chances for 
9 guessing conectly. But if there is a request for justification, a two-choice set 

quite acceptable. (Now you do not have a strict multiple-choice test.) 

7. Interview a range of students of the sort for whom the test is designed to find out 
how they interpret the questions and why they gave their answers. Revise test 
items that are revealed to unfairly help or hurt some students. 

8. Remember that wridng multiple-choice critical thinking items is an art. 

9. Careful scrutiny of items and multiple revisions are usually needed. Do not C5q>ect 
to make a good test in one trial. 

# Rules for ucm Writing 

1. Construct each item with one and only one correct or best answer. 

2. Avoid "none of the above" and "all of the above" as choi when examinees are to 
choose the best, rather than precisely correct, answer. 

@ 3. Use either a direct question or an incomplete statement as the item stem. 
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4. Write items in clear and simple language. 

5. State the central problem of the item clearly and completely in the stem. * 

6. Include most of the reading in the stem. 

7. Base each item on a single, central problem. 

8. Construct options homogeneous in granunatical form. 

9. Include in the stem any words that would othenvise need repeating in each option. 

10. Emphasize negative words or words of exclusion (e.g., "not", "except") and avoid 

such words when possible. ^ 

11. Place options at the end of the item stem not in the middle of it. 

12. Anange the options in a logical order, if one exists. 

13. Make all options plausible to examinees who do not know the correct or best ^ 
answer. ^ 

14. Avoid unintended hints based on: 

a. grammatical consistency or inconsistency between the stem and the options, 

b. repetition of key words in the stem and keyed option, or 

c. rote or other verbal associations between key words in the stem and the keyed ^ 
option. ® 

15. Avoid hints based on the: 

a. unusual length of the keyed option, 

b. degree of qualification stated in the keyed option or use of terms such as 
"never" and "always" in the unkcyed options, 

c. lack of independence and mutual exclusivity of the options, ^ 

d. frequency with which the keyed option is placed in a given option position, or 

e. pattern of the location of the keyed position. 

16. Avoid hints from one item to another. 

Suggestions f or Specific Aspects of Critical Thinking 
Judging the Credibilitv of Sources and Observations 

1. Have a comprehensive and defensible set of criteria forjudging credibility. 

2. Give a situation to which credibility criteria can be applied. 

3. Construct items so that only one criterion applies or most plausibly applies. 

4. Ask for comparative, not absolute judgments. For example, ask which, if cither, of 

the sources is more credible on a topic; or ask which, if either, of two conflicting ® 
observations is more credible. 
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5. Ask for a justification of the answer, if there is time for an cvaluator to read it. 
Realize that now the test is not strictly multiple-choice. 

Identifying Unstated Assumptions 

1. Distinguish the conclusion, pejorative force, and basis senses of "assumption". 

2. Provide an argument or an explanation in which unstated assumptions are made. 

3. Ask for '"omparative. not absolute judgments. For example, give several choices and 
ask "Which is probably taken for granted, though not stated?" 

4. Make sure no alternative is a conclusion that someone might draw from the stem. 

5. One and only one of the alternatives should complete, or strongly contribute to 
completing, a deductive connection from the reasons to the conclusion, or from the 
explanatory material to the fact to be explained. 

6. Ask for a justiflcation of the answer, if there is time for an evaluator to read it. 
Inducing and Judging Inductions 

1. Provide a situation with a conclusion to be judged on the basis of its being the 
best explanaticij of ihr. facts. 

2. Ask for comparative not absolute judgments about the conclusion. For example, ask 
either: 

a. which of two conclusions, if either, is more justified; or 

b. for the direction of support of the evidence (counts for. counts against, 
counts neither for nor against) rather than whether the conclusion is 
true, probably true, probably false, or false, or provided with insufficient 
data to go cither way. 

3. Ask for a justification of the answer, if there is time for an evaluator to read it. 
Deduction 

1. Provide an argument. 

2. Absolute judgment is all right here. For example you can ask whether the 
conclusion follows necessarily, or contradicts, or neither. Or you can ask. "Which, 
if any. of these follows necessarily from.^?" 

3. For more sophisticated students make sure that there are some items in which 
nothing follows necessarily. 
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Writing Multiple-Choice Critical Thinking Item:J (part 2) 
Stephen P. Norris 



Dr. Norris began by stating multiple-choice tests of thinking skills face a number 
of limitations. In such tests, examinees do not have to provide reasons for the answers 
they choose. If we do not know the reasons why examinees respond as the> do, then we 
are not able to justifiably conclude whether their responses are due to good or poor 
thinking. 

At the same time, multiple-choice tests have advantages. They can be scored 
relatively easily, they can test a large number of aspects of critical thinking in a short 
period of time; and they can thoroughly cover a single aspect of critical thinking much 
more efficiently than any other form of test. 

Dr. Norris asserted that by applying specific procedures during the design of the 
tests which build in the sort of relation we desire between good and poor thinking and 
keyed and unkeyed answers, respectively, we can preserve the advantages of both. 

This test building activity was demonstrated by discussion of development of the 
Test cn Appraising Observations.by Norris and King (Handout 1). It was emphasized 
that this test concentrated on only one aspect of critical thinking, judging the credibility 
of observatiomw 

The first step in designing a test is to define the domain of what is to be tested. 
In the example for this session, knowledge and use of a set of principles of observational 
appraisal were being evaluated. The Principles for Appraising Observations were 
included in the handout and used as a framework for participants as they looked at the 
development of specific test items. 

Trial items were then written to test various principles. The problem is how to 
determine if the items are good? Dr. Norris stated that an item would be good if when 
examinees thought in accord with the Principles, they chose the keyed answers, and if 
when they did not think in accord with the Principles, they chose unkeyed answers. 

To use Norris' method for determining if an item is good for a specific test, the 
developer should choose a sample from the entire population to be tested. (Such a 
sample usually cannot be random, but you should strive for reasonable 
representativeness.) Each examinee should be asked to read the items, choose an answer 
and then tell the developer all they were thinking in making their selection. Ideally, all 
probes should be open-ended. But reticent examinees can be asked more leading 
questions which seek justification or specific information. Everything examinees say 
should be tape-recorded. 

Based on their responses, examinees are given two scores: 

1. Performance Score: 

For each item, 0 = item answered wrong, 
1 = item answered right. 
For entire test, total number of right answers. 

2. Thinking Score 

- assessed independently from answer chosen 
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- For each item, 0-3 according to specific 
criteria (this scale can vary from test-to-test) 

- For entire test, total for each item. 

These scores can then be compared across all subjects for each item and thus 
provide an indication of the level of evidence for the validity of each item. The 
development of Evidence Scores using the scores above was discussed. This process was 
illustrated with specific examples from .he development of the Test on Appraising 
Observations. 

To accumulate Evidence Scores for each item over a sample of students, Norris 
developed a "Thinking/Performance Index Score" for each item and this was explained. 

Dr. Norris summarized by saying that this method was a combined qualitative- 
quantitative approach to the validity of a test. The hope is that in usinf this design 
methodology we can capitalize upon the strengths of multiple-choice tests and minimize ^ 
their weaknesses. 
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Taken from: Test on Appraising Observations Manual* Stephen P» Norris 5 Ruth King* St. 

John^s, Newfoundland: InstitSite for Educational Research and Development, 
Memorial University of Newfoundlands (Copyright, I985) 



TABLE A 
Principles for Appraising Observations 



L Observation statements tend to be more 
believable than inferences based upon them 



II. An observation statement tends to be believable 
to the extent that the observer 

1. IS functioning at a moderate level of emotional 
arousal. 

2 IS alert to the situation and gives his or her 
statement careful consideration; 

3. has no conflict of interest; 

4 IS skilled at observing the sort of thing observed. 

5, has a theoretical understanding of the thing 
observed; 

6 has senses that function normally. 

7. has a reputation for being honest and correct. 

8. uses as precise a technique as is appropriate, 

9. IS skilled in the technique being used; 

10 has no preconceived notion^ about the way the 
observation will turn out. 

11 was not exposed, after the event, to further 
information relevant to describing it, 

(If the observer was exposed to such information, 
the statement is believable to the extent that the 
exposure took place close to the time of the event 
described.) 

12 iS mature 



III. An observation statement tends to be believable 
to the extent that the observation conditions 

1 provide a satisfactory medium of observation. 

2 provide sufficient time for observation. 

3 provide more than one opportunity to observe. 

4 provide adequate instrumentation, if instru- 
mentation IS used 

(If instrumentation is used tn gaming access, 

then the statement tends to be believable to the 

extent that the instrumentation 

a^ has Suitable precision; 

b has a suitable range of application. 

c IS of good quality, 

d works in a way that is well understood, 
e IS in good working condition.) 



IV. An observation statement tends to be believable 
to the extent that the observation statement. 

1 . commits the speaker to holding a small number 
of things to be true. 

2 IS corroborated, 

3 IS no more precise than can be justified by the 
observation technique being used. 

4 IS made close to the time of observing; 

5, IS made by the person who did the observing. 

6. is strongly believed to be corrobora table by the 
person making it, 

7 doss not conflict with other statements for which 
good reasons can be given, 

8 IS made m the same environment as the one m 
which the observation was made. 

9 IS not about an emotionaliy-loaded event, 

10 IS the first report of the event provided by the 
speaker; 

1 1 IS not given in response to a leading question. 

12 does not report a recollection of something 
previously forgotten, 

13 reports on salient features of an event. 
(Features of an event are salient to the extent 
that they are extraordinary, colourful, novel, 
unusual, and interesting, and not salient to the 
extent that they are routine, commonplace and 
insignificant.) 

1 4 IS based upon a reliablv* record, if it is based upon 
a record 

(If an observation statement is based upon a 

record, then the statement tends to be believable 

to the extent that the record 

a was mede close to the time of observing 

b was made by the person who did the 

observing. 

c comes from a source having a good reputation 
for making correct records ) 
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Annotated List of Critical Thinking Tests 
Prepared by 
Stephen P. Norris 
for 



Assessing Higher Order Thinking Skills: 
7ssuefv J Practices 

TV K-^H. K.-^ionui Educational Laboratory' 

October 1 and 2, 1987 



In compiling this list I do not intend to give specific endorsement to any of the tests. 
The state of the art in critical thinking testing is rather more primitive than desirable. 
Thus, each listed test suffers deficiencies and some tests are worse than others. However, 
they are essentially all that is available. 

For guidance on choosing a critical thinking test I recommend the items under 
"References on Critical Thinking Testing." More specifically, I suggest the follo\^ing 
guidelines when choosing a critical thinking test: 

L Pay close attention to the directions, the test items, and the scoring guide. 

2. Take the test yourself and satisfy yourself that the scoring guide is 
reasonable for the students you wish to test. 

3. Ask yourself whether the test really tests for critical thinking. 

4. Ask yourself whether the test covers those aspects of critical thinking tl »t 
you wish to assess. 

5. Read the test manual looking for evidence on the validity and reliability of 
the test. 
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The Cornell Qass Reasoning Test, Form X 

R.H. Ennis, W.L. Gardiner, R. Morrow, D. Paulus, & L. Ringel. (1964). Illinois 
Critical Thinking Project, University of Illinois, 1310 South Sixth Street, 
Champaign, IL 61820. (Grades 4-14) 

The Cornell Conditional Reasoning Test, Form X 

R.H. Ennis, W.L. Gardiner, J. Guzzetta, R. Morrow, D. Paulus, & L Ringel. 
(1964). Illinois Critical Thinking Project, Universit>' of Illinois, 1310 South 
Sixth Street, Champaign, IL 61820. (Grades 4-14) 

Cornell Critical Thinking Te::t. Level X 

R.H. Ennis and J. Millman. (1985). Midwest Publications, P.O. Box 448, Pacific 
Grove, CA 93950. (Grades 4-14) 

Cornell Critical Thinking Test. Level Z 

R.H. Ennis and J. Millman. (1985). Midwest Publications, P.O. Box 448, Pacific 
Grove, CA 93950. (Advanced or gifted high school, college, adult) 

The Ennis-Weir Critical Thinking Essay Test 

R.H. Ennis and E. Weir. (1985). Midwest Publications, P.O. Box 448, Pacific 
Grove, CA 93950. (Grade 7-college) 

Judgment: Deductive Logic and Assumption Recognition 

E. Shaffer and J. Steiger. (1971). Instructional Objectives Exchange, P.O. Box 
24095, Los Angeles, CA 90024. (Grades 7-12) 

Logical Reasoning 

A.F. Hertzka and JP. Guilford. (1955). Sheridan Psychological Services, Inc., 
P.O. Box 6101, Orange*, CA 92667. (High school, college, adult) 

New Jersey Test of Reasoning Skills 

V. Shipman. (1983). Institute for the Advancement of Philosophy for Children, 
Test Division, Montclair State College, Upper Montclair, NJ 08043. (Grades 4- 
college) 

Ross Test of Higher Cognitive Processe s 

J.D. Ross and CM. Ross. (1976). Academic Therapy Publications, 20 
Commercial Boulevard, Novato, CA 94947. (Grades 4-6) 

Test on Appraising Observations 

S.P. Norris and R. King. (1983). Institute for Educational Resesrch and 
Development, Memorial University of Ne\^foundland, St. John's, Newfoundland 
AlB 3X8. (Grades 7-14) 

Test of Enouirv Skills 

BJ. Fraser. (1979). Australian Council for Educational Research Limited, 
Frederick Street, Hawthorn, Victoria 3122, Australia. (Grades 7-10) 

Watson-Glaser Critical Thinking Apprai.sal 

G. Watson and E.M. Glaser. (1980). The Psychological Corporation, 555 
Academic Court, San Antonio, TX 78204. (Grades 9-adult) 
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References on Critical Thinking Testing 

Arter, J.A. and Salmon, J. (1987). Assessing Higher Order Thinking Skills- A Consumer's Guide . 
Northwest Regional Educational Laboratory, Evaluation and Assessment, 101 S.W\ Main Street, 
Suite 500, Portland, OR 97204, pp. 15-31. 

Ennis, R.H. and Norris, S.P. (In press). Critical Thinking Testing and Other Critical Thinking 
Evaluation: Status, Issues, Needs. In J. Algina (Ed.), Issues in Evaluation . New York: Ablex. 

Norris, S.P. (1986). Evaluating Critical Thinking Ability. The History and Social Science Teacher . 
21, 135-146. 

Norris. S.P. and King, R. (1984). The Design of a Critical Thinking Test on Appraising 
Obser\ations . St. John's, Newfoundland; Institute for Educational Research and Development, 
Memorial University of Newfoundland. (ERIC document Reproduction Service No. ED 260 083) 

Walsh. D. and Paul, R.W. (1986). The Goal of Critical Thinking: From Educational Ideal to 
Educational Realit^^ American Federation of Teachers, 555 New Jersey Avenue N.W., 
Washington D.C. 20001, pp. 40-44. 
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^ Taken from: Test on Appraising Observations* Stephen P. Norris & Ruth King. St. John*s, 
Newfoundland: Memorial University of Newfoundland, Institute for Educational 
Research and Development. PART A (copyright, I983) 

A Traffic Accident norris 

HANDOUT 

A traffic accident has just occurred at an intersection which has a stop sign m each direction. 
Several cars were involved. 

A policeman and a policewoman will question people. Later several investigators will collect 
information about the accident. It is your job to judge the evidence given in the statements that 
follow. 



1 . A policeman is questioning Pierre and Martine. They were in their car at the intersection but 
were not involved in the accident. Martine is the driver and Pierre, who had been trying to 
figure out which way to go, is the map reader. 

The policeman asks Martine how many cars were at the intersecuon when the accident 
occurred. She answers, "There were three cars." 

Pierre says, "No, there were five cars." 

2. A small boy and his father had been standing on the sidewalk when the accident occurred. 
The boy says, "There was a motorcycle at the intersection." 

His father says, "No, the^e was no motorcycle at the intersection." 



3. A policewoman has been asking Mr. Wang and Ms. Vernon questions. She asks Mr. Wang, 
who was one of the people involved in the accident, whether he had used his signal, 

Mr. Wang answers, "Yes, I did use my signal." 

Ms. Vernon had been driving a car which was not involved m the accident. She tells the 
officer, "Mr. Wang did not use his signal. But this didn't cause the accident." 



# 4. The policewoman then points to Ms. Rosen's car which was one of the cars invol in the 

accident. She asks whether Ms. Rosen had signalled. 

Mr. Dawe, another driver not involved in the accic-^nt, says, "Ms. Rosen signalled. I was just 
talking to Ms. Vernon about this and I'm sure she will agree with what I said." 

Martine says, *'Ms. Rosen did not signal. I'm sure I'm right." 

5. The policeman talks to Mr. and Mrs. Peters, who were also involved in the accident. It is easy 
to see that Mr. Peters, who was the driver, is very upset by the accident. The policeman asks 
him to estimate his speed just before the accident. 

Mr. Peters says, "I was going about 15 kilometers an hour." 

A little later when he is feeling well he says, '1 was going about 30 kilometers an hour." 



6. The policeman asks whether or not the Peters' car had stopped at the stop sign. Ms Vernon, 
who IS a driver education instructor, says, "I am ve.'-y experienced in these matters. The 
Peters' car did not stop." 

Martine, who overheard this conversation, goes up to the officer and says, "The Peters' car 
did stop at the stop sign. " 



® 



29 

31 



NORRIS 
HANDOUT 

7. The officer turns to question Martine and Pierre and Mr. Dawe. The officer asks them to 
estimate the speed of Mr. Wang's car when it hit the others. 

Mr. Dawe says, "It was going about 40 or 45 kilometers an hour." 

The officer says, "It was going faster than that, wasn't it?" Martine says. "Oh yes, it was 
going about 60 or 65 kilometers an hour." 

8. Martine adds. "Mr. Wang went right through the stop sign/' 

The police officer turns to Mr. Dawe and says that at the scene of the accident Mr. Dawe 
couldn't remember whether Mr. Wang had stopped at the stop sign or not. Mr. Dawe says. "I 
remember now. Mr. Wang did stop at the stop sign." 



9. Ms. Vernon then says. "I also remember that a fancy blue sports car went through the stop 
sign." 

Martine says. "A car with twin headlights went right through the stop sign." 

10. Martine says. 'Three cars collided at the same time. There was one crash." 

Ms. Vernon says. "There was more than one crash. It would be very strange for the three to 
collide at exactly the same time." 

1 1 . The police officers ask the people involved in the accident and the other drivers to come to the 
police station to make off icia I statements. At the station, the policeman questions Mr. Peters. 

Mr. Peters points to a drawing of the intersection and says. "Just before the accident 
occurred Mr. Wang's car approached the intersection from that direction." 

The police officer says to Mr. Peters. "Surely Mr. Wang's car came from a different 
direction.' "Oh yes. " says Mr. Peters, "it did come from a different direction. " 



12. The policeman turns to Mr. Dawe to question him. In the background they can hear a 
conversation between the other officer and some of the other witnesses. Some are 
discussing whether one of the cars went through a stop sign. 

Mr. Dawe says. "Mr. Wang and Ms. Rosen crashed into each other. I saw it happen. ' 
"Also. I remember that a car went straight through a stop sign, too." 

13. Nearby, the policewoman and Martine are looking at the drawing of the intersection. 

Martine says. "A short time before the accident everyone was driving normally." 

She continues. "Then there was a loud squeal of tires. Mr. Peters' car tumed quickly 
toward the fruit stand on the corner." 

14. The policewoman asks Mr. Dawe to tell in which direction Mr. Pete;s was travelling before 
the accident. Mr. Dawe says. "He was going toward Fifth Street." 

The policewoman looks at her notes which were made at the scene of the accident. At that 
time Ms. Vernon had pointed and said that Mr. Peters was going away from Fifth Street 
before the accident. 
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In-classroom, Informal Assessment of Students' Thinking Skills 
Kenneth Bumgarner 



This session provided several examples of how classroom teachers could assess 
their students' thinking skills with non-paper and pencil, non-multiple-choice 
measurement methods. Ideologically, most of the methods presented were based on a 
model by Art Costa, which suggests the following 12 key behaviors be considered as 
evidence that student thinking is improving: 



(1) Persistence 



(2) Overcoming Impulsivity 



(3) Listening to Others 



(4) Flexible Thinking 



(5) Metacognition 



(6) Checking Accuracy 



(7) Questioning, Problem 
Posing 



time and effort put into task 
developing internal locus of control 
planning ahead 
evaluating alternatives 
dialogical reasoning 
paraphrasing 

more accepting of ambiguity 

brainstorming 

inner dialogue 

monitoring, adjusting thinking 

wait-time rather than 
immediate feedback 

focusing on a goal 



(8) Drawing on Knowledge assimilate, then apply 



(9) Precision of Language 
and Thought 

(10) Using all the Senses 

(1 1 ) Ingenuity, Originality 

Insightfulness 

(12) Wonderment, Curiosity 



trying for more 

accurate, thought out expression 
e.g., draw watch face 
unexpected combinations 



enjoyment of problem 
solving 




Throughout this session the audience was actively involved in each of these 12 
behaviors through a series of exercises illustrative of what a teacher could be 
practicing in order to stimulate thinking skills as well as of what behaviors should be 
looked for. Thinking was discussed as a process rather than simply requiring the 
students to arrive at an answer. A teacher should be more interested in observing 
behavior when the student does not know the answer, thereby being able to obser\e 
thought processes. 

Several rather interesting examples were given throughout the session, in which 
audience participation was high. Analogy and strategy problems were presented, and 
techniques for responding to the students were demonstrated through use of the audience 
as a classroom. Some of the examples included several samples of "parlor games*' 
activities, e.g.: 



"It is true for butter but not for bread ..." 

"A person may go through the door but not the window", 

"I will gladly drink coffee but not tea"... 

What is the relationship being expressed above? 



Metacognitive thinking was discussed and illustrated and the importance of the 
affective domain to encourage the thinking process was emphasized. Using all of the 
senses in teaching thinking skills was illustrated several ways, e.g.: 

*Try to draw (from memory) as close as possible the face of your watch." (I 
personally left the stem out of the picture and thought that I had four numbers 
which were actually only markers; my neighbor remembered quite a different set of 
characteristics.) 

A "Thinking Log" was distributed at the end of the session, which can be used 
to stimulate immediate active processing to Dromote retention and the construction of 
knoN\ ledge as well as a record keeper for the sessions within a conference. Portions of 
the "Thinking Log" are attached. 
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THINKING ABOUT THINKING 
CONFERENCE 
SECOND THOUGHTS 



SPONSOR: 
CO-SPONSORS: 

S '■■ < , o! fihuMhi' £.)i> .111 I'V 
<.ri\-^i 0:M'ir:'- New Ho'i/on>. 

If- •'• •(! J SloK- F^CiliWtu' — NiiliU'W 

D"' .vo'' NP^vo^K A^^(XiflliO'i f)' 

v. • hiti.|tf 'I Associ.iliO'i o' Sf'ino 
AiimirivMiofv U'^sriingion Ai^oCwIiOn 
,1* D .ifis Ditpctor U<i.hi iLjiufi 
Sl^f" A'.iOCi^IiOa (01 Super viiion nnd 

EdJf^l'On Associiil'on Wi»\hingior- 
RO'jntHDic \A -liriinqio^' Use Schoo' 
DifPclOn AsSOf-.lliOn 



Dear Participant: 

This pacl<et models a component of effective thinking- 
immeai:»te active processing. Current research indicates 
that retention is enhanced and the understanding of 
concepts is clarified when such processing is employed 
as a learning strategy. The packet provides a structure 
within which you can reflect on and synthesize the 
information gained during the conference. The thinking 
log is a place for you and your team members to record 
and process your ideas, thoughts, questions and plans. 

Please bring this log to each session. A block of time 
will be provided at the end of each session to allow you 
to analyze and record the essence of the presentation. 
Your notes will serve as a resource when you meet with 
your district team throughout the conference. 

Additional "Reflection" Sheets have been provided in the 
back of this packet for the use of conference participants 
who are attending preconference and/or postconference 
workshops. 



Jill Jacoby, Ph.D. Ken Bumgarner, Ed.D. 

Director of Staff Director, Basic Education 

Development/Curriculum Superintendent of Public 

Educational Service District 113 Instruction 
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1st Genoral Session 



A. List as many key words and phrasos as you can recall about this 
presentation. 



B. What three points do you consider most important. 
1. 

2. 

3. 

C. List three questions this presentation raised in your mind. 
1. 

2. 

3. 
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DISTRICT ANAI YSIS: Al TFRK|/^TIVF OMF 



If. like many districts, you are just beginning and have not yet started to 
work on thinking skills curriculum, you may wish to use this sheet to determine 
the current state of thinking skills in your school district. 



r 



How do you or does your district define thinking skills? Brainstorm 
and discuss suggested definitions. 



Briefly describe how thinking skills are currently taught in your 

classroom, school or district. Indicate whether it has been infused across 
the curriculum. 



3. What are some of the barriers to beginning or enhancing a thinking skills 
program in your area? 



4. What are some strengths your district brings to the task of establishing 
a thinking skills program? 



5. Other 
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If your district has worked on implementation of a thinking skills 
program over the last year, using your Reflection sheets as a resource, 
brainstorm the following questions to analyze your thinking skills focus: 



1 . Some important things we've learned over the past year In our 
school, classrooms, and district about teaching thinking. 



2. The strengf s of our present approach to Incorporating thinking 
skills into classrooms and/or curriculum areas are: 



3. Some additional components we should consider incorporating 
into our thinking skills program are: 



4. In our district, training in thinking skills should focus on: 



5. Other 



37 

39 



Qi 

DISTRICT PLANNIMG 
November 18, 1986 



BUMGARKER 
HANDOUT 



• 



Assuming that your district is interested in pursuing continuing a thinking 
skills program, briefly summarize your committee's recommendations as 
to how your district might proceed. 

Step 1. 

Step 2. 

Step 3. 

Step 4. 

Step 5. 

Step 6. 



^1 



Use the next sheet to help you map your plans. 
What materials/resources will you need? 

Estimate the cost of your proposal. 
Materials 
Consultant Fees 
Teacher Inservice 
Teacher Release 
Total 
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CALENDAR 



Start 



Finish 



OBJECTIVES 
Describe tasks 



PERSON/S RESPONSIBLE 
Name and phone number of 
Persons responsible for each task 
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Sow's Ears Into Silk Purses How to Take Unlikely-Looking 
Textbook Material and Make It Testable for Critical Thinking 

Connie Missitner 

Ms. Missimer's presentation was based on the premise that we should use the 
structure of "theory*' as our framework (theory) out of which to teach and assess critical 
thinking. Critical thinking is the making and appreciating of good theoiies or 
arguments. Arguments, theories, hypotheses, problems v/ith their solutions, even sales 
pitches--all have the same basic structure or framework. Ms. Missimer believes that it 
is the use of alternative theories which drives the intellectual life and causes progress in 
all subjects. 

Missimer presented a visual way to represent theories, using the simple drawing 
of a house. The ground is the problem, the walls are the reasons which support the 
roof, which is the conclusion, assumptions are below ground In the basement. Just as 
there are many houses on a street, so there are many possible solutions to a problem, but 
each must be supported with reasons. By helping students to see the structure of their 
arguments, and allowing a person to change arguments (build a new structure), we can 
incorporate critical thinking vocabulary, skills, and dispositions into all parts of the 
curriculum where teachers invoke this structure. 

Missimer showed how a play such as "Romeo & Juliet" can be much more 
interesting for students if they are offered to theorize about whether the play is any 
good, rather than to memorize ways that it is good. In the latter case, students are 
memorizing one theory at best, and are not considering alternative hypotheses. The 
latter is conducive to independent, critical thinking, and besides is more enjoyable. 

American Government textbooks are a challenge, since they are often made up of 
separate bits of information. Missimer indicated that even these texts can be used by 
teachers and students to create theories or arguments. For example, high school seniors 
could grapple with the question "What political institutions are missing from the table of 
contents?" (Hint: What political institutions are present in other countries, but don't 
appear in ours?") As students see that our political institutions don't include religious 
leaders, the military, etc., they come to understand the significance of the American 
system much better in contrast to other systems (which are, of course, theories). It is 
this "steroscopic" view of political systems, or indeed of any subject, which alternative 
theories provide. 

Missimer also showed an example of a test of higher order thinking which she 
designed. It is being used in a few high schools and colleges, although no technical data 
about its efficacy are available. The first section of the "Test of Critical Thinking Skills" 
offers arguments of increasing complexity. The multiple-choice questions ask "Which 
sentence is the conclusion?" "What reason(s) is offered?" "What is the issue?" The 
attempt here is to ascertain how well students understand an argument when it is 
presented to them. The second part consists of unfinished dialogues by two opposing 
parties. The student is asked which statement best continues the dialogue. (Statements 
other than the correct one are either contradictory or irrelevant.) The third section of 
the test offers a description of four major types of evidence: experimental, correlational, 
speculative, and observational. Students are asked to tell which type of evidence they 
should most expect in four different subject areas. 

The "Test of Critical Thinking Skills" is meant to improve on current tests 
(Cornell Critical Thinking Test and Watson-Glaser Critical Thinking Appraisal) by 
avoiding having students make inferences that are not directly spelled out in the 
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passages. It also does not require prior academic knowledge. Like Cornell and Watson- 
Glaser, this test does not assess how creative a critical thinker the student is. 

Missimer's test and handouts are attached. 
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^ Test of Critical Thinking Skills 

PARTI 

You will be asked to identify the Issue, condusion, and reason(s) in the 
« following arguments. The conclusion is theS niain, overall point being made. 
The reason or reasons are the ideas in support of the conclusion. TTie issue 
is the question about which the argument takes a stand. Please 
remember that in an argument the conclusion is not necessarily 
^ the last sentence. 

Driving while low on gas is a bad idea. Rrst, dirt particles at the 
bottom of the gas lank will be pushed into the motor with tiie remaining 
gas. Second, there is a danger of running out of gas, eitiier because tfie gas 
guage is inaccurate or t^cause the driver forgets to fill up the tank. 

1 . The conclusion of the argument is: 

® a) dirt particles at the bottom of tfie gas tank will be pushed into 

the motor 

b) tiiere is a danger of running out of gas 

c) driving while low on gas is a bad idea 
^ d) tiie driver forgets to fill up the tank 

2. A reason offered to support the conclusion is tiiat 

a) there is a danger of running out of gas . 

b) nmy drivers are careless 

® c) driving while low on gas is a bad idea 

d) gas costs less now than it did in the recent past 



3. The issue being argued is 

a) whetiier driving while low on gas is a bad idea and if so, why 

b) whetiier drivers are careless and if so, why 

c) whether gas costs less tiian ft did in tiie recent past and if so, 
why 

d) whetiier dirt particles get pushed into the motor and if so, why 
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Stealing is wrong. But the most important reason that it is wrong] is 
not what one might think, for instance that the person stolen from has 
been unjustly deprived of some things. Rather, stealing is wrong mainly 
because of the Injury that it does to the character of the thief. 



4. The conclusion of thjs argument Is: 

a) that the (^rson stolen from has been unjustly dsprived of some 

things 

b) that stealing injures the character of the thief 

c) that stealing does harm 

d) that stealing is wrong 



5. The reason supporting the conclusion is: 

a) that the person stolen from has been unjustly deprived of some 
things 

b) that stealing injures the character of the thief 

c) that stealing does harm 

d) that stealing is wrong 



6. The issue being argued is 

a) whether the person stolen from has been unjustly deprived of some 

things and If so, why 

b) whether stealing injures the character of the thief and if so, why 

c) whether stealing does harm and if so, why 

d) whether stealing is v^ong and if so, why 
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"Large-scale organization does not always diminish the individual's 
freedom- in some respects it enlarges it 

As a result of large-scale organization modem man enjoys freedoms 
that he could not hope to enjoy otherwise. Out of the vast and elaborately 
organized medical center come findings that free the individual from 
illnesses that have plagued mankind for centuries. The great urban 
university, which strikes some critics as little more than a huge factory, 
places within reach of millions of low-income worf^ers the opportunity to 
sunnount ignorance and stretch their horizons. 

The man who moves from a small town to a large dty experiences 
unaccustomed freedom. He not only escaf^s the stulti^ng web of 
attitudes, expectations and censorship that characterize the small town, 
he.finds in the city more choices in every dimension- kinds of dwelling, 
consumer goods, entertainment social companions, culture and worie" 

John Gardner, Self-Renewal, p.76 



7. Which of the following conclusions would go directly against 
Gaidner's point of view? 

a) Laige-scale organizations invariably diminish the 

individual's freedonL 

b) There should be a larger pool of consumer goods from which to choose 

c) Culture and woriq:lace are more interrelated than has previously 

been suspected 

d) Fewer illness^ plague mankind now than in past centuries. 



8. Which of the following are major reasons which Gardner offers for 
his conclusion? 

a) big urban universides axe little more than huge £actories; 

many people have moved to the city 

b) vast medical centers firee individuals from illness; big urban 

universides offer the poor a chance to leam 

c) people in small towns arc unaccustomed to the freedom of big cities; 

society needs more kinds of dwelling, consumer goods, 
entcrtaiimient, social companions, culture and work. 
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Here's the dilemma: Is it better to go to the market odd: a week or more 
often? Going once a week saves time and gas; on the other band, going 
more ofttn enables one to have iiesher vegetables and to take advantage 
of more sales. 

9. The issue is 

a) whether it is better tt) go to the market once a week or nx>re often 

b) whether going once a week saves time and gas 

c) whether going more often enables one to take advantage of moie 

sales 

d) the issue is not made clear 

10. The conclusion is 

a) it is better to go to the market at least once a week 

b) it is better to go to the moket only once a week 

c) it is better to go to the nriaiket more than once a week 

d) the conclusion is not made clear 

IL The reasons are 

a) all in favor of going to the markrt only once a week 

b) all in favor of goint the market more than once a week 

c) two reasons favor going once a week and two reasons favor going 

more often. 

d) it is not clear what the reasons are 
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The group, "First Amendment Rights for Children," makes the argument 
that schools should not censor books for children. A child should De able to 
go to the school library and request any book he wants, since children are 
citizens and therefore should enjoy all reasonable constitutional rights. 

One counterargument has been raised that school libraries should 
censor books for children. Son^ books should not be made available to 
children because they could be harmed by not understanding the full 
context of these books, take Truman Capott's In Cold Blood , for instance. 
This is an account of two men who murder a family. Capote is trying to 
understand the killers, but a child ought believe tha! he is advocating 
murder. Therefore children ought not to be allowed to read a book like 
(his until they are in high school. 

This counterargument is, hov/ever, incomplete. It does not take two 
i^^)ortant matters into consideration. First, grade schools have a limited 
amount of time to instill basic skills and ideas into students. Therefore, 
grade school administrators should not order any books that in their 
judgn^nt are a waste of time , le., do not help to insdll the basic skills and 
ideas that children need to learxL Second, schools don't have unlimited 
funds for school libraries, and so should order only those books which are 
instructive and enjoyable to children. Schools, therefore, should censor 
any books that in their judgment do not conduce to the best expenditure of 
time and money. 

12. The conclusion is that 

a) there are many arguments involved in the question whether 

grade school libranes should censor children's books 

b) censorship of children*" books by school libraries is justified 

c) children's books should not be censored, because the First 
Amendmwnt guarantees them the right to read what they wish 

d) grade schools are not doing a good enough job censoring students' 

reading material 

13. The reasons are that 

a) children could t?, harmed by not undent^ding the context of 
some books. 
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b) the child only has a limited amount of dme to absorb basic skills and 

ideas 

c) the school has only a limited amount of money for library books 

d) all of the above 

e) none of the above 



14. The argument assumes 

a) diat children need not have the same rights as adults 

b) that children have the same right to informed consent as adults 

c) that school authorities are unfair 

END PARTI 
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PARTn Continuing Dialogues 

The following is an unfimshcd dialogue between two people. You will be 
asked to anticipate a further remark tiiat would be consistent with the 
person's remarks so far. 

Hal: There's nothing to do here on campus except go to class. 

Mollie: ITl have to take issue with you, HaL Tliis week alone there was a 
jazz concert, a gallery show and three lectures by visiting professors. 

Hal: I see that I should have drawn a distinction between those activities 
you rightiy point out are extracurricular, and those things which I 
enjoy; Fve never seen the latter on campas. 

Mollie: Granted that you Ve never seen notices for extracurricular activities 
that you know you enjoy— is it possible that you arc falsely 
assuming tiliat you would not enjoy the activities we now have? 
How do you know you won't enjoy some of these school activities 
unless you try them? 

IS* Whsch of the foUowing could Hal say to further his line of 
reasciiing ^d remain consistent with what he has said so 
fai? 



Hal: a) Well, there ^ire many e.:;tractiaicular activities on 

.campus that interest me. I jcs^: haven't had the time to attend 
them. 

Hal: b) I don't believe thrl I am miiidng a felsc distiinction between 
activities I consider fun and those which the school supp ox ts. 
In fact, friends who have attended some of these school activities 
have told me about them, and Tm fairly sure they would bore 
me. 



Hal: c) lamoperatingontheassimiptiontiiatif you set your mind to 
it, you can enjoy anything. And such is tiie case witii school 
activities, whether they be of an extracurricular or academic nature. 
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Likewise, follow ibis new dialogue carefully so that you can add to 
Anne's line of reasoning* 

Anne: Students should have to take basic courses in all major subject areas. 

Brad: I disagree. Forcing students to take certain courses destroys their 
natural incentive to leanL 

Anne: But ifs claimed that most people will change jobs seven times 
during their lifetime. If colleges don't prepare students for a 
diversity of occupations, then they arc failing in tiicir duty to 
students. 

You arc assuming that four years of college arc supposed to prepare 
a student for all contingencies. I think that &e purpose of college 
is to get students excited about learning ia the hope that they will 
rctimi to college throughout their lives for morc education. 

16* Which of the following would logically extend the dialogue? 

Anne: a) You are assuming that making students take subjects will 
prevent learning excitement 

Anne: b) You are assuming that students should take some of 
the same subjects before graduation* 

Anne: c). You are assuniing that if students have to take certain courses 
they will become enthusiastic about subject areas which 
they would otherwise probably never know about 



Brad: 



END PARTE 
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We get evidence to support conclusions through these four means: 
sdentific experiment, correlation, speculation, or direct observation. 

1. Scientific experiment - 

Sdentific method involves setting up an experiment as well as 
the control (an almost identical situation, except that the 
variable to be tested in the experiment is missing.) 
- Sdentific method produces the strongest evidence, but it 
wori<s only if all variables can be conyolled. 
When that is not possible, the next best evidence is correlation. 

2. Correlation - 

With a correlation one can see that there is a relationship 
between two variables. The evidence that one variable is the reason 
for the other (condusion) is not as strong as in scientific method. 
But since we cannot hold down all variables in many situations, a 

correlation is the best evidence we have. 
3. Speculation- 
When we cannot set up a scientific experiment nor find any 
correlations yet still believe a theory to be true, we must justify 
our belief by an appeal to others' reason and experience. This is 
the area of speculation. 
4. A single observation - 

Sometimes the best evidence we can obtain is an eye-witness 
account. 



Any of these types evidence could be obtained badly and thereby be 
faulty or fraudulent. But assume here that whatever the type of evidence 
which could be obtained, it would be of the highest quality. With that 
presupposition in mind, which of these types of evidence would be the 
strongest that you could expect in the following: 

Example: 

A theory that v/hen people exercise they decrease their chance of 
a heart attack. 

aexperimr ' b. con'elation c. speculation d. a single observation 

The corrsu v is b. It would not be ethical to hold people in 
C;ages and ores some to exercise, others to rest. So experiment is 
out. The ^^e'jt we can do is to correlate the incidence of heart 
attack to the amount of exercise in a large nurnijer of people. 
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Choose among the four types of evidence- 
experiment, correlation, speculation, a single observation— 
and determine which of these would be the strongest that 
you could expect In the following circumstances: 

17. A theory that war is never Justified. 

a experiment 

b. correlation 

c. speculation 

d. a Sifigie observation 

18. A theory tiiat plants do not grow in response to sound, 
a experiment 

b. correlation 

c. speculation 

d. a single observation 

19. A theory that when the dollar drops in value, tiie stock nwket 

rises. 

a experiment 

b. correlation 

c. speculation 

d. a single observation 



20. Atourt case that >c^in individual committed a murder. 



END PART III 
AnswGrst 

Part I 1c. 2a, 3a, 4d, 5b, 6d, 7a, 8b, 9a,10d, 11c, 12b. 13d 140^. 

Part II 15b, 16a, 

Part Ill17c,18a,19b,20d 




Test prepared by CA Missimer, c 1987 

52^3 



MISSIKER 
HANDOUT 



P^gogical Sctiem#s for Critical 
TMiiking: 

How do yoQ pnll it out of tozis? How caa yon it? 
• For the conference, "Assessing Higher Order 

Thinking Skills/ Oct 2, 1967 
By Connie Missimer 

9 First briefly, vrtiat is critical tMolcing? 

I would claim that critical thinlcing Is tns molclsig ond 
appreciating of good arguinents. 
Synoityms f or orguing: hypothssizlng, theorizing, thocaw that. 
^ Unlike narratives, which proceed by a temporal thread, 

arguments are structures (see diagram, last page) 



What specific Icinds of questions will proiuco critical tliinking ? 
® '^aostiosis tliat liold up competing tlioori@s or argnmonts. 

1. Arguo that theory I OC) is proforablo to theory Y vY). 

e Argue that equality of opportunity is preferable to equality of results. 

Argue that Darwin's theory of evolution is preferable to Lamarck's. 
Argue that Uving in 17th century France is preferable to living in 
the U.S. today. 

Twil me why the boy in the story should have stayed with the Pooh 
® Bear instead of going right home. Now tell me why the boy should 

have gone right home. Now, what do you think he should have 
done? How come? (fun-fun versus fun-serious reasons 
Tour esample(8) 

9 



or 

Fmthor tho argomont that the te:st is making that I is pref erdble 
to Y; Of opposo that argument 



2. Argue for and against 
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o.g., Arguo for and against the i<iea that Japanese oom|>etition is a 
decisive factor in the slowing of U.r economic gro^ in the 'SO's. 
Give me reasons \^ should or should not have a spelling test 
tiiiswsek. 



3. Croat© a t&at would o^Mst 

e^.. Create a theory t^t would e^iain \fhy Milton's most attractive 

character in Taradise Lost" is Lucifer, 
or Create an idea of v^y tSie girl could have lost her hat (An reason 
John gives Incomes "John's theory" in furtJier di^nission.) 
7ou want students to hunt for a variety of reasc-ns. 

4. What conclusions could you draw about (e^-^dy's view of life, stellar 

motions, the economic outlook) from the^ 

foatares in Hardy's novels: harsh landscapes, lonely people, etc. 

p&^omeiaa recently sighted at Palomar: 

recent ©v^^ts in tlie U.S. economy: 

Or If toe author of the book says that , what 

do you think he means (or is concluding)? 

Since I found papers on the hall floor, a banana peel on the front 
stairs, and spilled juice in the bathroom, what could I conclude 
at>out your tidiness? (And you hope for a rai^e of ansvi^s so 
you get to say) And which conclusion/inference is more 
justified, in your opinion? 

The featur*^, phenomena, or events the book or you cite are tiie 
reasons for your students' and your tlieoretical conclusions. You want 
tliem to play aroxmd with various inferences. 
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4. (harder) What sigaificsnt fact (vain©) is missmg in 



e.g.. What value was missii^ from popular culture in the 

U.S.intlie 1970's? Make an argument about the implications of 
the lack of this value. Or Create an argument that this value is 
(is not) central to a healthy society. 



Or What could we in the UnitfKi State have considered before we 
started using a lot of electricity? 
When Marian got mad and hit Joey, v^t oth'^r \«ray of behaving 
do you think might have t>een missing from her mind? 

Questions that are not immediately conducive to critical 
thinking (but \^ch may be quite important for other 
purposes!) 

Can you summarize, list> name the 

Just "compare and contrast" do not do the full job, because they do not 
offer reasons, just conclusions. 

Questions which might be conducive to CI, but which seem 
ambiguous in that r^ard: 

What four x's are involved in process y? 

What is the significance of 

Disc'dss and its relevance to 



Connie Missimer 
4636 HE. 40th St 
Seattle, WA 96105 
(206) 522-4965 
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V fultuuitig a list uf ftrcjUtnih ubcd phtubCb in uiguinctiianun, 
h bonic illustrative examples of iheii Ubt. 

ret to disagree An aiiutii lli.ti di^uciik .lu .u .m iiu|iJA>c ami ^iiuuldn't 
kii'paiguiDg. prolrjiilt tHcaust iIk\ ar|*iic inacirik. Argiict.Nbliuiild 
ugret tu cii^agtcc when U) ittet need to |;et ttiote (aciit;il evidtitic. ii tin* 
aiguniriii liitiged on a fact, oi (2) ilie\ mtd loioiiic up %%idi dincicm 
uppmachrs or expanded ar^uiutnis. 

umptioQ The reason beneath ihc slated rcabun(&}. Liihti the auihcr will 
nm Slate them at all. or ihe author will announce ihtm b\ saying. "I am 
assuming that,** or "The assumpium here is/* 

sed on the premise thai *'Bastd on the reason (or assumption) thai.** 

\cd on the presupposition that Some as 'b^^cd ou die pfcutise that/* aU»\c. 

:ause hidicairs lhai a conclusion has piecedtd and a reason sill follun. 

M Seechaptei 12. 

.im An) staietneiu. Claims art provisional, subjtci to change with new 
infotmaiion. 

juce Toarriic ai a patticular instante thai is cnuilcd in a gcneitfli^aiiun. 

Ofitn luund in the foun. "ironi winch wt ma) deduce ihai.** "All out 

parachute^ open nhen the rip cord is pulled, frum v%liich yuu may deduce 

ihai ihe one you bu) iVom us will open too.** 
rinttion Ail that is meant to be included in a term, 
tinction (Beuvecn two terms) An Important difference between two 

terms. 

iw- the inference that Come to the conclusion that. 

tw the conclusion that Same as **draw the inference thai.** 

!Ogrant«ng thau*. still In a counitrargutnem. a technH|ue for acccpiing or 
granting the truth of tlu* other person*^ reason(s), and yet showing how 
the) are consistent uiih ytmr argument. **E\en gt anting that it is frighten 
ing to speak out in a group and that people ma) think you're stupid, stiil it 
is better to speak \'our mind because the world needs your ideas just as 
much as it needs anybody else*s.** 

%t dichotomy between In a counterargument, the claiir that a false 
distinction has been made. He: "In college. )'ou cpn either have a lot or^'un 
or you can do a lot of studying.** She: "Vou have creased a false dichotomy 
between ha\ing fun and studying. B) George. )'0u can have a loi of fun 
studying!** 

ages on Usually the argument hinges on. or is dependent on, some outside 
fact or other argument (e.g.. "The aiSgumeni that most societies have 
suppressed women hinges on the claim that there is no biological 
expiattaiion foi won)en*s faiiL^re to hold positions of power**). 

.•then What folloits "iP is the reason; what follows *'ihen** is the conclusion 
to that statement. 

plications Consequences beyond the conclusion. Either the author does not 

state them or the author says. "The implications are.** 
piy Most frequenti) found in a counterargument or used b) a respondent 
in a deliberation. It is a request to clarify someone eise^ conclusion. ''Vou 
seem to be suggesting a connection between the cultural \atues held b) a 
^ groi;spand their a\erage income Are )'Ou implying that most wealth and 
poveri\ are the result of these values?" Or )ou can use it to say what you 
don^t mean* **Bul I don't mean to imply that.** A less frequent but handy 
use; One idea netessarily invohes antnher (e.g.. '*Urama implies conflict j. 

•'tv«t 'I'tii* ihiint^ :» |fi*ii, t:i! niii^Kinn finni :i \t^\f n »s<ii'iif i*v O 
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Leaps ^re necessar\ but Mime leaps ore safer than others. UsualK used to 
describe an unsafe one: **In suggesting that. ..on the baMS ol just a (ew 
cai^es. the author has made tmi great an inductive leap.** 
Infer Come to the conclusion that. "From the louk of his si^cai) bud) one can 
infer that he has been doing btienuoiis exercise.** or "We can make the 
inference that.** 

Inference The. conclusion. *'Alihough he thinks it*s right to conclude that we 
should be patriotic, he doesn*t think it right to base this inference on the 
premise that we are superior to people of cHher nations.** 

Issue The topic or subject at hand. 

Make the argument Same as "one could argue that.** "One could make the 
argument that.** 

Necessary c^use A fanor without whicli some effect would aoi occur. ''0^)gen 
is a necessary cause of fire** (no ox)gen, no fire). It Men takes se\eral 
necessary causes to create an effect (e.g.. to ^et a rocuet into space, or to 
get to work on time). See abo Sufficient cause. 

Necessarily Must be the case (e.g., "necessarily involves**). 

Not necessarily I>oesn*t ha\e to be the case. *'BiC«!use some people are belter 
at argumentation doesn't neces>arii) impi) (indicate) that the) are 
smai ter. The&e better arguers ha\e probabi) just been practicmg longer", 
often found in the phrases, "dches not necessaril) impI).** "is ncH necessarily 
the case." 

Ought Usuall) signals the solution a thinker has arrived at u% a result of his 
conclusion. **Because of the rapid rate at v;hich we*re polluting this lake. I 
conclude that we will not be able to swim in it by 1995. So we ought to find 
a new lake to pollute (just kidding).** See aUo Should. 

Play the devillft advocate Take the opposite pcHnt of view. Student to instruc- 
tor. *Td like to play the de\ii!i advocate to )t>ur position that our class 
should have an in<lass final. Think of all the extra lo\e and attention we 
could give to your questions if we had lots of time to mull over them at 
home.** 

Premise Reason. See abo Based on the premise that. 

That presupposes That requires the assumption (underl)ing leason) that 

**That argument presupposes that eveiyone is born with e^ual enthusiasm 

and talent.** 

Reason The support for the conclusion, the why. Write. *The reason is.** don*i 
write. -The reason is because.- because that^ redundant. The reason for 
her tardiness is her laziness.** or The reason that she is often tard) u that 
the bus schedule is erratic.** 

Should Signals a solution or a prescription (as opposed lo £ sutement about 
what the case is). See abo Ought 

Sufficient cause A cause that b) itself will create an effect. **Oxygen is a ~ 
necessary cause of a fire, but it isn't sufficient. On the cnher hanc^lack of CT C 
oxygen is a sufficient cause of choking to death.** ' ' ^ 

Therefore Signals that a conclusion must follow and that a reason came befc.e ^ ^ 
(usuall) right before, but not always). *'She^ always up at 4 a.m. Thereiore, > m 
shels dead tired by 10 r.M.- § cn 

Warranted inference A phrase to the effect that a conclusion and/or the step a g 
from the reason(s) to the conclusion is justified. "Man) studies ha\e found § 
a positive correlation between the amount of time students spend on a 
course and their grades. These studies warrant the inference that sheer 
effort expended is a central factor in doing well in Khool.** 
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Prescription? 
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Definitions? 
Distinctions? 



Conclusion 



J/ 



Consistency? 



\Evid 



ence 



Issue: whether... 



Assumptions 



Implications 

Alternative 
Arguments-^ 

Inference 
Warranted? 

Reason(s) 
True? 



The Structure of Argument 



_C.A. Missimer, Good Argument'' 
(cJPrentice-Hall, 1986 

■il 3U J^C '-j'-' 
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How To Select A Test Of Higher Order Thinking Skills 
Judith A* Arter 



The goal of this session was to promote good consumerism by discussing the features 
that tests of HOTS should have, and comparing currently published assessment devices to this 
ideal list. The purpose was not to criticize any particular tests or imply that we should not use 
multiple-choice tests, but, rather, to describe assessment issues and concerns so that users know 
the strengths and weaknesses of what they are using. The list of features was taken from two 
appendices Assessing Higher Order Thinking Skills; A Consumer's Guide published recently b> 
NWREL. These appendices were --eproduced as a handout for this session and are attached. 

Basically, the checklist for selecting tests of HOTS reflects the issues and concerns that 
were raised during the keynote address and responses by the panel. These issues, and how the> 
relate to current tests are: 

A. Content. The first concern is to select a test that looks like it measures what you want 
it to measure. Not all tests of HOTS attempt to measure the s?ime thing. 

1. Tests can have vastly different emphases. The first four columns of the attached 
chart describe the content of the tests. There were tests found that covered critical 
thinking, problem solving, development (e.g., Piaget), creativity, structure of the 
intellect and Bloom's Taxonomy (e.g., achievement tests). 

2. Even when tests are called the same thing, what they cover can be different (e.g., 
Watson "Glaf;er v. Cornell Critical Thinking Tests). 

3. Does the test measure HOTS in an atomistic or holistic manner? Most tests are based 
on taxonomies of HOTS where each question is designed to measure one skill. There 
are few th9.t look at how all the skills work in concert. 



Are the questions abstract or is the attempt made to have them reflect real-life 
situations? Some tests (e.g., the structure of the intellect tests) can have questions 
which are very abstract e.g., analogies, number sequences, syllogisms, and pattern 
recognition. The rationale for the item types needs to be clear, and needs to be what 
you want to emphasize. 



5. 



One problem is that the more "real" a situation becomes, the less easy it is to have 
questions with only one right answer. Abstract items generally have only one clearly 
right answer. 

Is the test contem embedded in a subject area or is it designed to be ^'subject free?" 
Examples were p**nvided of situations in which the claim of the test to be embedded 
in a subject area werf; not entirely accurate. 



6. Is the test based op some theoretical model? Most of the tests are. However, as was 
pointed out during the opening session, not all theoretical models are equally good. 
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B. Usefulness. 

1. Is the instrument or method easy to use? Most tests are group, multiple-choice tests. 
These are generally easy to use. Some measures are individually administered or 
open-ended. These tests are generally much harder to give and score. So, content 
advantages that might arise from having more of an open-ended test are balanced b> 
having them harder to use. 

Column 8 of the chart provides information about test format; columns 9 and 10 gi\e 
information on how long it takes to administer the test. 

2. Is there help with interpreting the results and using the results for instructional 
planning? Columns 11 and 12 of the chart provide information on these points. In 

general, the tests are lackl g on this dimension. There are some that are tied directl> ® 
into curriculum materials. 

C. Reliability. Test reliability refers to the consistency with which a test measures what it 
does. Column 13 provides iniTormation abcut the reliability of various tests. In general, 
the total-test score reliability of most tests is reasonable. However, subtest score 

reliabilities tend to be very low. ^0 

D. Validity. Validity is a key issue. What evidence is there that the test measures what it 
claims to measure? The test should provide evidence that scores are due to HOTS and 
not due to extraneous factors. Below are li:5ted some of the studies that ideally would be 
done on a test. Column 14 of the chart discusses validity. 

® 

1. There should be a effort at content validation. Part of this is reviews of the 
literature and relationship to a theoretical model. Another aspect might be review of 
questions by knowledgeable judges do judges agree that the questions measure the 
skills claimed? Most tests have adequate review of questions. 

2. Is the te:;t measuring HOTS or vocabulary? The students need to understand what is 
being asked or the test does not measure HOTS. Most tests do not see if this is the 
case. 

3. Right answers are arrived at only through the thinking process being claimed and not 

for other reasons such as differences in general knowledge, cultural differences or © 
content area knowledge. The presenter illustrated these points with examples from 
several tests. Most tests do not examine this. 

4. Correlations are presented between test scores and scores on other measures, and 
there is some rationale for why the correlations presented support the validity of the 

test. Many tests report correlations but do not explain why they support its validity. ^ 

5. What is the evidence to support the subtest structure of the tests. Many of the tests 
claim to measure separate subskills. Are they really different? A factor analysis is 
sometimes done to examine this issue. The structure of the intellect tests are 
generally best at this because that is where the structure comes from. 

6. Is there other evidence of validity such as logical group differences in scores or 
differences in scores after instruction? These are also not usually done by tests. 




7. Does the test discuis these measurement issues and huw it has attempted to deal with 
them? Most tests do not even acknowledge that they exist. Two tests that are good 
in this regard are the Test of Appraising Observation and the Cornell Critical 
Thinking Test. 



In general, there are many assessment issues regarding HOTS. Some of these are equall> 
relevant to instruction definitions, reductionism, and the degree to which they are embedded 
in content areas, ^^jrrent commercially available assessment devices are of variable quality. 
Test developers are, however, continuously attempting to attack these issues as new tests are 
developed. 
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HIGHER ORDER THINKING SKILLS TESTS 



101 SW Main Str««t, Suitt 500 
PortUnd, Ongon P7204 

T«Uphont (503) 275 WOO From. Agt— Inn Hightr Ordtr Thinking Skillg, A Contumer't Cuid^ by Judith A. Artor und J«nnif«r Salmon, PorUand, OR. 

Northwett Regional Educational Laboratory, 1087. 
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Subject 



Test 


Focus 


Grades 


AREAS 


Score Obtained 


AVAILABILITY 


ACHIEVEMENT T 

Assessment of 
Re^iing Growth 
(1980) 


ESTS: 

Inferential 
Comprehension 


3.741 


Reading 


Inferential 
comprehension score 
available 


Jamestown Publishers 
P.O. Boi 6743 
Providence, RI 02940 


C^ifomia 
Acbievemcot 
Test (1985) 


Bloom's 
Taxonomy 


K-12 


Reading 
Language 


HOTS score re-$cored 
from other subtests 
by publisher 


CTB/McGraw Hill 
2500 Garden Road 
Monterey, CA 93940 


Comprehensive 
TesU of Basic 
S5cills (1981) 


Bloom's 
Taxonomy 


K-12 


Reading, Math 
Language, 
Science, Social 
Studies, Ref. Skills 


Items are cross- 
referenced to Bloom's 
Taxonomy. User must 
generate HOTS score. 


CTB/McGraw Hill 
2500 Garden Road 
Monterey, CA 93940 


Iowa Test of 
Basic Skills 
(1985) 


Various 
depending on 
subtest 


K-12 


Listening, 
Reading, Maps, 
Ref. materials 


Same ms above. 


Riverside Pub. Co. 
8020 Bryn Mawr Avenue 
Chicago, IL 60631 


Metropolitan 
Achievement 
T«is (1985) 


Bloom's 
Taxonomy 


K-12 


Reading, Math 
Science, 
Social Studies 


HOTS score re-scored 
from other subtests 
by publisher 


Psychological Corp. 
555 Academic Court 
San Antonio, TX 78204 


National Tests 
of Basic Skills 
(1985) 


Inference 
Evaluation 


PreK-adult 


Reading 


Inferential and 
evaluative comprehension 
aggregated by publisher, 
% correct on individual 
objectives. 


American lesnronics 
P.O. B^»« 7770 
Iowa City, lA 52244 


Reading 
Yardstick? 


Interpretation 
Evaluation 


K^ 


Reading 


interpretive and 
evaluative reading 
available from publisher 


Riverside Pub. Co. 
6020 Bryn Mawr Avenue 
Chicago, IL 60631 


Scan-Tron 
Reading Tests 
(1985) 


Inferential 
Comprehension 


3^ 


Reading 


Items are cross- 
referenced to 
skills. User must 
generate score. 


SCAN-TRON Corporation 
Reading Test Division 
2021 East Del Amo Bv. 
Rancho Dominiquei, CA 90220 


SRA Achievement 
Scries (1985) 


Various, 
depending on 
nibtest 


K-12 


Reading 
Math 

Social Studies 
Science 


Items are cross- 
referenced to 
skills. User must 
generate score. 


Science Research Associates, Inc. 
155 Wacker Drive 
Chicago, IL 60606 


SUnford 
Achievement 
Test (1981) 


Using 

information 


1.9 


English 
Science 
Social Studies 


Using informatiou 
score, re-fcored 
fn»n other subtests 
by publisher. 


Psychological Corp. 
555 Academic Court 
San Antonio, TX 
^204^2 


Stanford Test 
of Academic 
Skills (1982) 


Using 

information 


S-adult 


English 
Science 
^Dcial Studies 


Using information 
score, re-fcored 
from other subtests 
ty publisher. 


Psychological Corp. 

555 Academic Court 

San Antaiio, TX 78204^32 


Survey of 
Eksic Skills 


Various, 
depending on 
subtest 


K-12 


Listening, 
Ri;:iing, Math 
Social Studies 
Science 


Users consult the 
skills profile 
report for each 
subtest 


Science Res&^rch /Xssociates, Inc. 
155 Wadcer Drive 
Chicago, IL 60606 


The Three R's 
Tests (1982) 


Logic^l 
relationship 
Literary analysis 


K-12 


Rc&ding 
LsagTja^e Arts 
Mathematics 


Item analysis 
scores available 
from publisher 


RiversiJe Pub. Co. 
8420 Bryn Mawr Ave. 
Chicago, IL 60631 



Author's pi^rposc 
problem solving 
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Interpretation of Table Codes 

Norms (Value judgement implied) 

None No normative information is provided 

Eak Has some standards of comparison, e.g., means of research sample, 
decile norms or item statistics. 

Good Has norms based on t good sized sample gi lots of other 
information. 

Excellent Has norms based on a national sample, and other information. 
Other Interpretation (No value judgement as to the quJity of the assistance is implied) 

None No help with interpretation provided. 

Some Has some help with inteipretinR scores, e.g., what the various 
scores mean. 

Some-f Has information on what the scores mean and some help with use 
in inst'-uction. 

Extensive Has extensive information on what the scores mean and how to 
use them in instruction. 

Reliability (Value judgement implied) 

None provide d No information was found. 

Poor All r's below .70 

Fair At least one reported r is greater than .70 

Good Total r is greater than .85; most subtests have r greater than .75. 

Excellent Several kinds reported; total score r is greater than .90; most 
subtest scores greater than .80 

Validit\ (This describes the quantity of information available, not necessaril> the extent to 
which the instrument is valid.) 

No information No information on validity is reported. 

Some information At least one activity related to validation is reported. 

Some-f information Validity was examined in several different ways. 

Extensive infornr ition Special effort was made to examine validit> and there is a large 

research base on the instrument. 
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Checklist for Selecting a Higher Order Thinking Skills Test 

Usefulness 

A. Information Obtained 

1. Do the stated uses of the instrument match up with what you want to use the 
information for? 

2. Does ♦he instrument or method measure the HOTS skills on which you want 
information? 

3. Does the instrumeat assist with interpretation of results? Does it have criteria 
by which to judge results? This includes statements about what performance 
should be like at various grade levels. It could also include norms. 

4. Is there information about how to use the results to plan instruction for 
students? 

B. Logisticb 

1. Is the instrumem or method easy to use? 

2. Is it easy to score and interpret the results? 

3. Is the length of time required to collect information acceptable? 

C. Cost 

1. Are costs within available resources? (Include costs of obtaining the 
instrument or method, training data collectors and collecting data.) 

Technical Adequacy 

A. Theoretical Basis 

1. Do the supporting materials for the instrument or method present a clear 
definition of the aspects of HOTS that it claims to measure? Does the test 
manual discuss how this definition was developed and why the test has the 
content it has? Is evidence provided (based on research or theory) that the 
definition(s) and tebt content are reasonable? 

B. Reliability 

1. Was the instrument pilot tested? 

2. Is there some measure of reliability available for the instrument? 

a. For a structured-format test this includes at least item discriminations, 
internal consistency and test-retest reliabilities. 

b. For an open-ended te'it this would include estimates of reliability of 
scoring such as interrater reliability. 
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c. If the results are going to be used to make important (and hard to reverse) 
decisions about individual students, reliability should be above .90. For 
group uses, or for educational decisions that are easily reversible, 
reliabilities should be above .75. 

C. Validity: Is there evidence that the instrument measures what it claims to measure? 
Validity is in the relationship between the instrument and its use. Tliere 
should be evidence that the instrument can be validly used for the purposes 
stated. For example, what evidence is there that the item types used measure 
the skill area? 



1. For structured-format instruments an ideal set of validity studies would 
include: 

a. The respondent understands what is being asked. Vocabulary or concepts 
unfamiliar to a group would make the instrument unusable for that group. 
This information would most likely be obtained by observing or 
interviewing stude^nts. 

b. Right answers aie only arrived at through the thinking process claimed to 
be measured not from clues c: ''"ulty assumptions. Likewise wrong 
answers are arrived at through faulty reasoning and not due to good 
reasoning based on a different philosophical orientation or experience 
level. This information would most likely be obtained by observing or 
interviewing students. 

c. There is a moderate correlation with intelligence and achievement tests. 
Scores correlate with other validated tests claiming to measure the same 
thing. 

d. There is a factor analysis done to show that the subscales do measure 
different things. 

e. Groups that should be different in their scores are indeed different. This 
could include the ability of an instrument to differentiate between types 
of students. 

f. The instrument measures changes or differences in HOTS after training 
designed to change HOTS. 

g. There is a clear and frank discussion of the measurement issues involved 
including which aspects were investigated during the development process 
and which were not. 

h. It is the opinion of knowledgeable judges that the instrument measures the 
HOTS aspects claimed. 

i. For Piagetian instruments there is a high correlation between scores on the 
test and level of formal reasoning obtained from clinical interviews. 

2. For open-ended instruments this would include: 

a. The respondent understands what is being asked. Vocabulary or concepts 
unfamiliar to a group would make the instrument unusable for that group. 
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This information would most likely be obtained by observing or 
interviewing students. 

There is a moderate correlation with intelligence anc* achievement tests. 
Scores from the instrument correlate with scores from other instruments 
claiming to measure the same thing. 

Groups that should be different in their scores are indeed different. This 
could include the ability of an instrument to differentiate between types 
of students. 

The instrument measures changes or differences in HOTS after training 
designed to change HOTS. 

There is a clear and frank discussion of the measurement issues involved 
including which aspects were investigated during the development process 
. .d which were not. 

It is the opinion of knowledgeable judges that the instrument measures the 
HOTS aspect^ claimed. 



Diagnosing Thinking Deficiencies in the Classroom 



Selma Wassermann. 



Dr, Wassermann began by making a connection between the relatively new fields 
of teacher testing and assessing thinking skills. She feels that both are attempting to 
measure acts which are: 

-complex, sophisticated and interrelated; 

-observable and intuitive; 

-creative; 

-inconsistent; 

-independent and interrelated; 
-ambiguous and nonlinear. 

The hazards of assessing students thinking skills with paper and pencil tests are 
that the tests: 

-drive the curriculum; 

-produce linear thinkers; 

-lose intuitive, sloppy acts of geniuses; 

-have only a single score; 

-restrict thinking with forced choices; 

-label and categorire people; 

-take away important teacher judgments (observation); 
-discriminate against slower thinkers. 

In lieu of paper and pencil tests, Wassermann suggests we use a variety of means 
to assess thinking: behavioral scales, performance on tasks which use a variet> of skills, 
and teacher observation. The problem with these means is that they are high Inference 
and therefore "soft" data but done properly can be highly effective and relevant. 

Observation, performance and behavioral measures: 

-contribute to our overall understanding of the student; 
-provide the most valuable data; 
-provide long-range data; 

-look at thinking applied in many contexts as well as when it is not applied; 
-can be used for targeting thinking deficits in individual student instructional 
plans. 

The remainder of the workshop was a hands "^n application of Wassermann's 
"Teacher Rating Instrument" which has eight student "types" displaying different 
thinking deficits. Many of these deficits appear to be affective and would be improved 
b> j^jsitive self-esteem. Wassermann would disagree. She maintains that these are 
cognitive deficits. The instrument and two case studies are attached. 




ASSESSING THINKING-RELATED BEHAVIOR 
Selma Wassermann WASSERMAN 
Simon Fraser University HANDOUT 



Task 1: 

(1) Work in pairs. 

(2) Read through the eight behavior profiles in the 
Teacher-Rating Instrument. Talk with each other 
about any students you have obset^ed who exhibit 
these patterns . 

(3) Then, read the brief Case Study of Bob. 

(4) Talk with each other about what you perceive to 
be important behavioral indicators of Bob's 
deficits in his thiiking capability. 

(5) Use the Teacher — Rating Instrument and decide on 
how you will assess Bob*s behavior. 

(6) Follow the same procedures with the Case Study 
of Eddy. 



Task 2: 

(1) Work in pairs. 

(2) Read through the 33 behavioral indicators on 
the Profile of Student Perf orriancc . Talk with 
each other about students you have observed 
who exhibit these characteristics in their 
academic task work. 

(3) Then, re-read the Case Study of Bob. Discuss 
what you perceive to be any observable posit ive or 
negative indicators in his performance on the 
Comparing assignment . 

(4) Using the Profile of Student Performance, decide 
on an appropriate assessment of B"ob*6 ability to 
compare . 

(5) Follow the same procedure with the Case Study of Eddie 
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Selma Wassermann 
Faculty of Education 
Simon Fraser University 
©1986 
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Notes to the teacher: 

The teacher rating instruinent helps you identify students who 
are exhibiting behaviors associated with deficits in higher-order 
thinking capabilities. Used in pre- and post-assessments, it also 
provides means for examining positive gains in those behaviors 
associated with thinking. 

Read each profile. In section A, after each profile, write the 
names of thc3e students whose b( havior you beheve to be clearly 
described in the profile. In section B, after each profile, write the 
names of those students whose behavior may possibly reflect that 
described in the profile (i.e., you are not certain; there is some 
element of doubt). 
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1. Profile of the very impulsive student. 

These sbidents typicaUy act without thinking. When a problem 
or activity is introduced, these students leap into action first. They 
don't seem to have a plan, nor do they consider alternatives. The 
mode of operation is doing_- and sitting down to "think tilings out" 
does not seem to be ti\eir pattern of behavior. Impulsive students do 
not engage in reflection. ^ j 

Associated witi\ impulsiveness is the idea of rapid and random 
movements, of acts directed by little more tiian whim or caprice. 
Action itself seems to be much more important than thinking about 
possible modes of action. To be up and doing is so prominent in the 
behavior of tiiese students ti\at tiie purpose or goal of the doing is 
neglected. 

• A. This pattern of behavior soimds very much like: 



B. This pattern of behavior sometimes describes: 

1 3 

2 4 
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Z Profile of the very dependent student. 

These students typicaUy want help vnth practicaUy eveiything 
they undertake. They find it hard to begin work ^^thout asking for 
help in getting started. Once they have begun, help is again 
requested. Often such students say, "I'm stuck!" or "What shall I do 
now?" or "I don't know what I'm supposed to do." 

These shidents' insistent calling on the teacher for help is a 
strong characteristic of their inability to carry out tasks 
independently. When the teacher's help is not available, these 
students may just sit there and do nothing imtil help is forthcommg. 

A. This pattern of behavior sounds very much like: 

1 3 

2 4 

B. This pattern of behavior sometimes describes: 

1 3 

2 4 
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3. Profile of the loud, dogmatic and overly assertive student. 

These students seem to have "all the answers," and thsy are 
unyielding in their conviction that they are mhi. Such students 
reject discrepant data; their minds are made up. These students stick 
to their positions regardless of the facts. There is a quality of 
inflexibi^itv about the way they think. We generally consider them as 
intemperate, imreasonable and insensitive to the feelings of others. 
They don't listen to alternatives. In fact, they seem reluctant to 
acknowledge that there are alternatives. 

These students are apt to impugn the motives of those who 
oppose their views. Their language is frequently studded with 
extreme words, such as always, everybody, no body, never. They 
may generalize loosely about races or nations or religious groupings. 
We sometim cs think of them as authoritarian; that they try to 
dominate other people; that they are rash in their judgments, dose- 
minded, intemperate, o unreasonable in presenting their ideas. 



A. This pattern of behavior sounds very much Mke: 

1 - 3 

2 4 



B. This pattern of behavior sometimes sounds like: 

1 3 

2 4 
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4. PiTof ile of the rigid, in-a-rut student. 

These students typically want to stick to doing things in the 
same old ways. They don't like new or different ways of doing 
things. Tne fact that a problem or task is new and calls for new 
piocedures doesn't make a difference. These students try to force old 
methods onto new problems. When that procedure doesn't work, 
these students complain that the problem is at fault. 

These students are most comfortable when carrying out 
routines. They can learn lessons and formulas, but they have great 
difficulty in applying the prindp! i of what has been learned to new 
situations. 



^ A. Thispattemof behavior sounds very much like: 

1 3 

2 4 

B. This pattern of behavior sometimes describes: 

1 , 3 

2 4 



83 



80 



WASSERI-iAN 
HANDOUT 



5. Profile of the student who misses the meaning. 

These students don't seem to understand. They miss the point - 
- of a lesson, an assignment, a story, a joke. Such a student might 
say, "I don't get it." We think of these students as people who "don't 
listen" and who don't pay attention. Actually, the problem is that 
they are not able to interpret data intelligently or extract meaning 
from their experiences. It is as if their ability to process data has 
been seriously impaired. 

A. This pattern of behavior soimds very much like: 

1 3 

2 4 



B. This pattern of behavior sometimes describes: 
1 3 

2 : 4 
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6. Profile of the student who can't concentrate. 

The dominant characteristic of these students is their tendency 
to use means that are inconsistent with or inappropriate to the ends 
they seek. It's not that these students don't have any ideas. They do 
have ideas about what tney want to do, but the paths that they take 
to arrive at their goals may be silly or illogical, impractical, or even 
irrational. There seems to be an absence of awareness in choosing 
these paths. Their choices do not reveal a great deal of thinking 
about the connections between means and ends, but ratiier suggest 
an indiscriminate and random selection of means. Perhaps that 
accounts for the fact that the goals these students set are seldom 
realized. 



A. This pattern of behavior soimds very much like: 

1 3 

2 4 



B. This pattern of behavior sometimes describes: 

1 > 3 

2 4 



S5 82 



WASSEK-IAN 
HANDOUT 



7. Profile of the underconfident student. 

These students lack confidence in expressing their thoughts. 
During class or group discussions, such students rarely volunteer 
information, not bece use they have nothing to say, but because of 
fear of exposing v^hat they are thinking to public scrutiny. At the end 
of class, these students might say privately to the teacher that they 
did have an idea, but v^ere concerned about how it might have been 
received by the group. Underconfident students are not necessarily 
shy; they rather lack confic ence in their ability to think and are 
fearful about exposmg theL' thoughts to possible criticism. 



A. This pattern of beha vior sounds very much like: 

1 3 

2 4 



B. This pattern of behavior sometimes describes: 

1 3 

2 4 
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8. Profile of the student who actively resists thinking. 

These students typically scorn thinking as a preferred mode of 
operation. They believe that action is more important and that 
thinking is for the "intellectuals." Classmates who are seen as 
intellectually advanced are held in contempt. When aslred to do some 
thinking, these students reject the process. "It's the teacher's job to 
tell us what to do" may be the response these students give. Such 
students see theix' function as acting rather than reflecting. They 
make strong value judgments about "people who think," and those 
judgments are clearly negative. 

A. This pattern of behavior sounds very much like: 

1 3 
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B. This pattern of behavior sometimes describes: 

1 3 

7 4 
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CASE STUDY OF BOB (A 8la:th grade boy) 



Bob's cumulative school records showed that he had 
difficulties in school from first grade on. His first 
grade teacher wrote that his work habits and general attitude 
were poor. "Although he has dona adequate work, I fcei he 
could do better if he wpald only apply himself," Notes from 
his second grade teacher read, "I am constantly after him 
to get on with his work. His work is not as good as it 
could be. He just isn't trying." From the third grade 
teacher: "While Robert appears to have the ability to do 
better, he is functioning on a below cverage level in all of 
his subjects. He is a big disturbance in class." Other 
teachers noted is "wandering behavior, bis lack of preparation, 
his tendency to procrastinate; that he was the last to finish 
his work." The fourth grade teacher wrote that "he finds 
it difficult to stay put for long; that he lacked the ability 
to stay on task." 

His sixth grade teacher notedi "He needs close super- 
vision and structuring. He seems unable to function indepen- 
dently. He seems unable to put his mind to a task^ instead 
of engaging, he avoids. He seems to need to conform and 
becomes very agitated when he is asked to think things out 
for himself. He expresses deeply held racist views and is 
very adamant about his point of vie^; and rarely allows that 
another opinion has validity." 

The following are some selected events th^t are typical 
of his classroom perfori ance. (It should be noted that Bob 
was considered to be of "normal" intelligence; that he had 
no other discernable "learning disability;" and that he 
suffered from no observable emotional disturbance.) 



tit 



When he was asked to do some planning for his 

social studies project, he said (to the teacher): 

" You* re the teacher. It's your job to tell me 
what to do." 

Hhen he observed that the classroom sink had become 

clogged he offered: "You know why this thing's 
clogged the plumbing is cheap." 

After the teacher had ber ^ absent for a day, he 
said, "Don't he absent anymore. I need you here. 
I need you to tell me what to do." 

In a discussion on the Hindus and Muslims^ he 
said, "What^s this about them believing that 
cows are sacred? ■ Cows runi.ing loose and being 
worshipped? Isn't that pretty stupid?" 



** Bob: "You 
and we eat 



know, 

nea t ; 



we eat . protozoa . Mee t 
so we eat 



has cells 



85 



protozoa 

88 



WASSERKAN 
HANDOUT 



- 2 - 



When asked to compare a news article with an editorial 
(he was given copies of each), he wrote: 

"My thought. As you can see, there is 
much difference between the articles." 
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CASE STUDY OF EDDIE (A sixth grade boy) 



9 



Eddie is an attractive boy; gregarious and appealing 
in many ways. Of "normal" intelligence, he has consistently 
been categorized as "under achieving." He is physically 
5.-althy and school records show nj discernable evidence of 
emotional instability. 

From early first grade experiences, Eddie was described 
as "quite immature; noisy, restless and annoying to other 
children." His third grade teacher noted that he tended 
wo rush through his assignments, maklug careless, thoughtless 
errors; that in his work, speed rather then accuracy seems 
to be his goal." The fourth grade teacher surveyed her 
year with him by writing, "He is easily diverted from any 
task^ As a result, he jumps from activity to activity with- 
out completing assigned material. He seems to need constant 
attention and constant encouragement. He cannot concentrate. 
I have tried to point out to him that we cannot accomplish 
a great deal by doing three or four things at one time; but 
this seems difficult for him to understand. He does not 
follow directions, written or otherwise. Naturally all 
this behavior uirectly affects his academic performance." 
Eddie^s fifth grade teacher indicated similar problems: "Eddie 
doesn't listen. He has difficulty following directions. 
This causes him to frequently do the wrong work. Often he 
misinterprets what has been said*" The sixth grade teacher 
noted: He has a great deal of difficulty doinp, independent 
work. His work is haphazard and impatient. He is most 
comfortable when following routines; if something new is 
proposed, he rejects it and asks for 'doing it the way we 
used to. * " 

The following are some selected events that are 
typical of his classroom performance: 

Eddie (to his teacher): Sometimes when you (teacher) 
©ay it''s all right to do our work in pencil, I start 
to use a pencil. Then I look at the guy uext to me 
and he is using a pen. The guy next to him is using 
a pen. So I ri p my paper and start all over 
again with a pexi . 

** Eddie: "I have a good idea for a science project. 
I want to do a project on st ic electricity with 
balloons. I did that last ye.., too." 
Teacher: ''How come you want to do it again?' 
Eddie: ^t was a gooi project last year and I 
want to do it again. 
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** Eddie: "I don't understand. Whr t are we supposed 
to do here?" 

** Eddie: "Just tell me what I'm supposed to do, I'll 
do it," 

*t Eddie: "I like to do things the way I've done t'uem 
before, I don't like new ways of doing things.' 

** Eddie: "When I get stuck, I look around for somebody 
to help me, I have a hard time getting started, because 
I'm not sure about what I'm srpposed to do," 
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PUPIL'S NAME 



BATE OF ASSESSMENT (1) 

(2) 



THINKING SKILLS DEVELOPMENT PROGRAM 

PROFILE OF STUDENT PERFORMANCE 
("How Well Does Sammy Think?") 



(c)Selma Wasserraann 
Faculty of Education 
Simon Fraser Univers 
1984 
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This 33 paired-item rating scale is designed to help the teacher 
diagnose a student's performance on tasks which require thinking at higher 
cognitive levelst 

Each of the paired items reflects a ''positive*' and a "negative" view 
of pupil behavior in the performance of thinking tasks. Since a student's performance 
may vary, your assessment on each of the items should reflect your impression of 
a student's overall functioning over a period of time, rather than on a single task. 
In making your diagnosis, use the following scale as a guide: 



Positive behavior 



Thinking-rv-iat ed behavior 



J L 



_! !_ 



-J L 



_!_ 



A ! L 



_1_ 



o 



This 

behavior 
is seen 
almo St 
all the 
t ime 



+3 
This 

behavior 
is 

frequently 
seen 



+2 
This 

behavior 
is seen 
somet imes 



+1 

This 

behavior 
is rarely 
seen 



-1 

Ihis 

behavior 
is rarely 
seen 



-2 
This 

behavior 
is seen 
somet imes 



-3 

This 

behavior 
is 

frequently 
seen 



-A 
This 

behavior 
is seen 
almost 
all the 
t ime 



.^4 



V hat do the ratings tell us ? 

Ratings which are consistently on the "minus" side of the scale would reveal 
that a student is having considerable difficulty with thinking skills. A student with a 
high "minus" ocore would be manifesting extreme "thinking-related behaviors" -- that is, 
b^ihaviors associated with inadequate cognitive functioning. Higher negative ratings 
indicate that student needs much more work on the development of thinking skills. 

Ratings which are consistently on the "plus" side of the scale point to 
students who are exhibiting successftil thinking skills development. Higher positive 
ratings point out the gifted and talented thinkers, who are likely to benefit enormously 
from add it tonal work on t h in king tasks. 

This diagnostic assessment should provide the teacher with an analysis of 
individual pupils* strengths and weaknesses in performing thinking tasks. The most 
important use to be made of this instrument is to enable the teacher to use each 
diagnosis in helping each student to further develop his/her thinking capabilities. 
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A. PUPIL PERFORMANCE OK OBSERVING TASKS 



Rating_s Positive behaviors 

1, To what extent does this student 
show that hie/her observations 
reflect careful and thoughtful 
attention to substance and detail? 



2. To what extent are the observations 
accurate? 



Thinking-related behaviors 

la. To what extent are the 
student's observations hasty 
and manifest of "Insufficient" 
thought? 



2a • To what extent does the student 
go beyond the data and make 
unsupported assumptions Ir his/her 
obser vat ions? 



Ratings 



3, To what extent Is the student 
discriminating In his/her observations? 



3a. To what extent Is he/she 
satisfied with superficial treat- 
ment of the material? 



Teacher's diagnostic observations of pupil's observing skills: 
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B. PUPIL PERFORMANCE ON CLASSIFYING TASKS 



Ra t Ings 



Positive behaviors 



ThinkinR-r elated behaviors 



Ra t ing 



4. To what extent does this student 
shov an ability to create groups for 
the categorization of objects? 



4a, To what extent does the student 
get "stuck" in the formation of 
groups? 



5. To what extent are the student's 
groups related to some common principle? 



5a . To what extent are the groups 
random, and unrelated to each 
other? 



6. To what extent is he/she able to 
place objects into the groups where 
they belong? 



6a, To what: extent is he/she 
unable to show discrimination in 
the sorting and classifying of 
obj ec t s? 



Teacher's diagnostic observations of pupil's classifying skills: 
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C. PUPIL PERFORMANCE ON COMPARING TASKS 



Rat Ings 



Positive behaviors 

7, To what extent do the com- 
parisons show careful and 
thoughtful obser vat Ions of 
substance and detail? 



8. To what extent do the compari- 
sons enable the student to arrive 
at deeper meanings about the Items 
be Iny compar ed ? 



9. To what extent are the 
comparisons comprehensive In 
scope? 



Thinking-related behaviors 

7a. To what extent are compari- 
sons hasty and rv^flective of 
superficiality? 



8a. To what extent are the 
compar 1 sons reflect ive of 
only the surface aspects of 
the items? 



9a. To what extent are the 
compar 1 sons reflective of 
"iiisuf f Iclent" thought? 



Rat ings 



Teacher^s diagnostic observations of pupll^s comparing skills; 
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PUPIL PERFORMANCE ON LOOKING FOR ASSUMPTIONS TASKS 



Rat Ings 



Positive behaviors 

10, To what extent does the student 
show that he/she Is aware of the 
differences between fact and 
a ssump t ion ? 



Think InR-r elated behaviors 

10a, To what extent Is the student 
"stuck" and unable to comprehend 
the concept of assumptions? 



Rat Ings 



11, To what extent Is the student 
able to Identify several assumptions 
underlying an event? 



11a. To what extent Is he/she 
satisfied with the finding of a 
single assumption? 



4^ 



12. To what extent does the student 
place value on the Importance of 
discriminating between ascumptlon 
and fact? 



12a . To what extent does he/she 
consider the identification of 
as sump t ions inconsequent la 1 ? 



Teacher'^ diagnostic observations of pupil's skills In lookln,^ for assumptions 



^0 
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£• PUPIL PERFORMANCE ON HYPOTHESIZING TASKS: 



Rat Ings Posit Ive behavior 



13. To what extent Is the student 
able to come up with several thoughtful 
and appropriate hypotheses for problem 
sltuat Ions? 



ThlnklnR-related behavior 

13a. To what extent Is he/she 
satisfied with a single response? 



Ratings 



lA. To what extent is the student 
capable of suspending judgment and 
keeping an open mind In the examination 
of alternative hypotheses? 



lAa. To what extent is he/she 
dogma t Ic about proposing solutions? 



l*^'. To what extent is the student 
comfortable with the cognitive d is 
sonanace underlying the tentative 
and provisional aspects of 
hypot heslz Ing? 



15a. To what extent does this 
kind of ambiguity make the 
stud en t uncomfortable? 



16. To what extent is he/she capable 
of performing hypothesizing 
activities using his/her own thinking? 



16a. To what extent is he/she 
dependent upon the teacher's help 
in carrying out the tasks? 



Teacher^ s diagnostic observations of pupil \s s kills in hypothesizing: 



C 'p 
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F. PUPIL PERFORMANCE ON IMAGINING TASKS: 



Rat ing8 



Positive behavior 

17. To what extent Is the strident 
able to create new and original 
schemes? 



18. To what extent does the student 
enjoy creating and imagining? 



19. To what extent does the student's 
imagining activities show a talent for 
creat iveness, inventiveness and 
or iginal ity? 



Thinking-related behavior 

17a. To what extent does he/she 
tend to repeat and draw upon 
forms which have been done 
many t Imes before? 



18a. To what extent is this an 
Intimidating activity for this 
stud ent ? 



19a. To what extent are this 
student's in.agining activities 
rout in e, standard and lacking 
in creative flair? 



Ra t ings 



Teacher's diagnostic observations of pupil's imagininR skills: 
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G. PUPIL PERFORMANCE ON DECISION-MAKING SKILLS 



Ra t Ings 



Positive behavior 

20. To what extent Is this student 
able to think through a situation 
and arrive at a decision? 



21. To what extent Is he/she able to 
take ownership for the decision made? 



22. To what extent Is he/she "In touch" 
with the values he/she holds which 
guide that decision? 



Thinking-related behavior 

21a. To what extent does this 
student get "stuck" and unable 
to come to his/her own decision? 



21a. To what extent does this 
student avoid taking respon- 
sibility for his/her decision? 

22a. To what extent does this 
student's decision seem to be 
made up on the spur of the 
moment and not guided by a 
clear and consistent value 
system? 



Rat Ings 



Teacher's diagnostic observations of pupil's de cision-making; skills; 
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PUPIL PERFORMANCE ON PROBLEM SOLVING SKILLS; 



Rat lags 



Positive behavior 

23. To what extent Is this student 
able to function on problem solving 
tasks on his/her own? 



T hinking-related behavior 

23a* To what extent Is he/she 
dependent upon others for help 
with each step of the problem? 



Ratings 



To what extent Is he/she able to 
gather data and process the data? 



2Aa* To what extent does he/she 
get stuck and bogged down, and 
unable to complete the process? 



25. To what extent Is he/she able to 
come up with possible solutions that 
reflect the data gathered? 



25a* To what extent are the 
solutions "made up" or guesses? 



ID 

oo 



26. To what extent Is he/she able to 
come up with several possible 
solutions? 



26a, To what extent Is he/she 
looking for just a single 
answer? 



Teacher ^8 diagnostic observations of pupll^s problem-^solvlng skills: 
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Ra t Ings 



ADDITIONAL PERFORMANCE CRITERIA FOR ALL THINKING TASKS ; 

Positive behavior Thinking-related behavior 



Rat Ings 



27. To what extent does the student 
give his/her full attention to the 
task of thinking? 



27a, To what extent does the 
student give only token attention 
to the tasks? 



28. To what extent is the student able 
to take cognit ive risks and give responses 
that show originality and creativity? 



28a. To what extent are the 
responses more standard, and 
more traditional in their 
approach? 



to 
to 



29. Vo what extent does the pupil 
show that he/she values the process 
of thinking? 



30. To what extent is the pupil able 
to carry the tasks to completion? 



29a. To what extent does the 
pupil pay more attention to the 
product — i.e. , getting the 
"right" answer? 



30a. To what extent does he/she 
leave the tasks unfinished? 



31. To what extent does the pupil 
show an understanding of the process 
of the operation behind each activity? 



31a. To what extent is the pupil 
responding in a rote, formula-like way? 



32. To what extent does the pupil show 
his/her ability to deal with situations 
where there are many acceptable answers? 



32a. To what extent does the pupil 
need reassurance that the work is 
"right" and "good?" 
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33. To whfit extent does the student show 
a high level of engagement of thought on 
the tasks? 



33a. To what extent does he/she 
avoid the th in king tasks because 
thinking is too hard? 



Teacher ^s diagnostic observations of overpll pypil performance; 
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Assessing Creativity 
Anita Halstead 



Creativity is a relatively new concept that Ms. Halstead defines as ''the ability to go 
beyond perceived patterns and rules and have value/' There is general agreement that the four 
skills that measure creativity are: 



- Fluency: ability to generate a lot of ideas; 

- Flexibility: ability to take many different viewpoints; 

- Elaboration: ability to extend an idea, add, refine it so it is better understood, 

- Originality: ability to produce statistically unique ideas. 



In addition there are issues of: sensitivity to problems, leadership, and humor. 

There are many paper and pencil creativity tests available but these kind of tests do not 
measure the "essence" of creativity. Halstead favors informal assessments, followed up with 
positive reinforcement (for those who do favorably) and action. Act on the results rather than 
file awa> for the future. Tests can give people knowledge about themselves. Most teachers and 
supervisors cannot predict who will do well on a creativity measure. 

Divergent questions can informally assess creative thinking: 



- Generate ideas: both within and outside of content areas; 

- Viewpoint: look at something from a different perspective ("how would an ant see 
this?"); 

- Involvement: affective; 

- Conscious self-deceit: (Hypothetical wishing. If I had...) 

- Forced association: how^ are two unlike things similar? How are two similar things 
different? 

- Reorganization model: looking at something in a different way; 



Ms. Halstead related several informal performance measures for assessing left and right 
brain dominance as well as asking us to solve a problem which we scored for fluency, 
flexibility, elaboration and originality. 

She maintains that it is possible to train people to think creatively by making them 
aware of the creative process. Children and adults are often the best judge of their own 
creativity. But when making formal assessments of children, it is important to use multiple 
sources of data and to be cognizant of children's developmental stages which are; 
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- Dreamer: 18 nionths-2 years; fantasy and nonverbal activity; 

- Poet: 3-6 years; rhyming, nonsense language (Dr. Seuss); playing with words 

- Inventor: 6-8 years; takes things apart; inventive; mixes materials; wants to know 
how things work; 

- Poet: 10-12 years and in 20's; plays with semantics; journal, diary writing; 

- Inventor: 12-16 years and 35-40 years; challenging and wondering why 
things occur. 

In general, a creative child has these qualities: IQ range of 118-130, controlling, 
dominant, unusual interests, high energy, adventuresome, temperamental, egocentric and 
disorganized. 



Assessing Higher Order Thinking Skills in Hawaii: One State's Experience 
Judith A. Arter and Lynde Paule 



The purpose of this session was to describe some work in progress that NWREL is doing 
under contract to the Hawaii State Department of Education. The project is to develop several 
sets of questions designed to measure various aspects of the State's learning objectives for 
students, called FPOs. 

One of the eight areas to be assessed is "develop decision-making and problem-solving 
skills." The seven subskills listed by Hawaii for this objective are shown on page 1 of the 
presenters' handouts. In order to clarify the subskills to provide guidance for item 
development, NWREL staff listed various item types that could be used to address each subskill. 

About 40 items were written to cover these skills. Dr. Arter related that the items were 
written with several goals in mind. First, the authors tried to make the situations in the test 
reflect real- life situations. Second, multiple questions were written for each situation so that 
the situation could reflect several pieces of information and so that the situation could be 
examined from several perspectives. After review by Hawaii State Department staff, 24 were 
selected for pilot testing. 

The procedure used for the pilot test, and the rationale for this procedure, are provided 
on pages 2-5 of the presenters' handouts. The pilot test was designed to address some of the 
validity issues that surround multiple-choice tests of HOTS. In a multiple-choice test the right 
answer is a proxy for the skill that the item is supposed to measure. That is, if a student gets 
the item right then the assumption is that the student has demonstrated the skill. Similarly, if 
the student gets the item wrong then the assumption is that the student has not demonstrated the 
skill --in this case good thinking. The general procedure was to have a group of students take 
the test orally read everything out loud, indicate words and passages that were not 
understood and provide the rationale for the answer they choose. If students typically ^et a 
question right or wrong for other than the intended reasons then the question will be »ltered or 
dropped. 

Twenty-eight students were tested in this manner. Each tape is from 70 to 120 minutes 
long. Two raters reviewed the tapes. The scoring system used to rate the rationale for answers 
is provided on page 6 of the handouts. In addition to scoring all distractors and right answers 
in this manner, the raters made lots of notes on each student in an attempt to adequately 
capture thinking processes. 

Although the data are still being analyzed. Dr. Paule related that the results relate both 
to items and to students. Problems with items included: 



1. Differences in the general knowledge that different students bring to the task. This 
influences several items. 

2. Differences in cultural values that affected one or two of the items. Specifically, 
there was an item than required the student to select the best solution to a problem. 
One student based his rationale on the value that agreement between people is more 
important than anything else. 
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3. Many of the passages and questions appeared to be at too difficult a reading level. 
Sometimes this indicated a need to change wording. Other times it indicated student 
deficiencies for example, knowledge of critical thinking terminology such as 
"criteria," "assumption," and "unbiased." 

4. A few times students came up with good reasons why the response we thought was 
right was not right, and why a distractor might be the better choice. 

The process may also have uncovered some deficiencies in student thinking patterns. 
"Ma> have** is important here since the sample is small, and because sometimes . hen students 
had extreme difficulty in providing rationales for answers it was hard to tell whether there was 
a problem with the questiuP ur a problem with the student thinking pattern. Some issues ^ixh 
respect to students that should be follov/ed up include the following. In the presentation, each 
of these points was illustrated with a portion of one of the student tapes. 



1. Some students sef*med to have difficulty putting themselves in someone else's shoes. 
Some questions required the student to pretend that they had a particular list of 
concerns and respond from that viewpoint. Students would often ignore this task and 
answer from their own opinion. 

2. Students often did not answer the question or refer back to the passage to answer the 
question. Rather they would answer each option as being a separate true/false 
question or drift off into wild flights of fancy as to why a choice is right or wrong. 



3. Many students seemed to have difficulty articulating why they choose the answer 
they did. 



4. As mentioned above, many students had a lot of trouble with thinking skills 
terminology. 
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PPO III Content Description 



0 



Al. Identifies, clarifies, and states a problem and develops criteria for 
examining alternatives in solving the problem. 



1. 


Identify central issue or problem 


Airbags, Pemba, 






Gorge, Cartoon 


2. 


Restate problem or paraphrase 




3. 


Compare similarities, develop analogies 




4. 


Develop criteria for judging 


Drugs 


5. 


Design an experiment 


Grades, Detergent, 






Lightbulbs 


Gathers information from various sources and analyzes and organizes 


the 


information to facilitate the formulation 


of alternatives. 


1. 


What other information is needed 


Pemba, Airbags, 






Detergent, Lightbulbs 


2. 


What information is relevant/irrelevant 


Gorge, Airbags 


3. 


Where to get information 


Pemba 


4. 


Fact V. opinion/biased v. unbiased sources Airbags 


5, 


Checking the consistency of information 


Drugs 


6. 


Identify assumptions 


Airbags, Detergent 


1. 


Appraising observations 


Gorge 


B. 


Which two values are in conflict 




Formulates hypothe5>es about a problem based on available information. 


1. 


Develop hypothesis 


Detergent 


2. 


Formulate questions that lead to deeper 






understanding 




Applies the criteria established to select an 


alternative . 


Evaluates the alternative selected for its effectiveness. 


1. 


Which proposed solution might be best 


Pemba 


2. 


Applying criteria 


Airbags 



A6. Draws conclusions or generalizations based on the alternatives or 
hypotheses and related information. 



1. Inferences and deductions Gorge 

2. Logical syllogisms Baboons 

3. Recognizing adequacy of data 

4. What information supports a conclusion 

5. Cause and effect 

6. Interpreting the results of an experiment Detergent 



Grades, Gorge 



A7. Validates and reports the Conclusions and modifications, if any. 

Pemba, Airbags 



1. What to do to validate choice 

2. Probable consequences 

3. How to report results 

4. Expanding results 



Detergent 
Detergent 
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PPO III Pilot Test Procedure 



Purpose 

FPO III questions are intended to measure higher-order thinking 
skills* The purpose of the pilot test is to try to determine whether 
they are fulfilling this function. Essentially our position is the same 
as Norris and King (1984): 

•The view guiding this study is that validity investigations of 
ability tests involve three steps: (i) finding out whether examinees 
understand the tasks on the test in the intended way; (ii) given that 
the appropriate understanding is there, determining whether examinees 
use appropriate approaches to complete the required tasks; and (iii) 
if number of correct answers is to be used as the indicator of 
examinees' ability, determining whether the keyed answers are 
justified given appropriate strategies for arriving at angers." (p. 
38) 

"The general principle of ability test validation underlying this 
study is that ability tests are valid to the extent that good 
thinking leads to good performance on the test and poor thinking 
leads to poor performance. The attempt is thus to explain 
performance in terms of thinking, and to do this there must be a 
description of the thinking processes which lead to performance. . . 
For a test to be suitably valid, there roust be at least an overall 
tendency for good and poor thinking to be linked to good and poor 
performance respectively.* (p. 39) 

Thus, if we want to show that the questions measuring FPO III are 
really measures of higher-order thinking we need to find out the thinking 
that goes into arriving at answers. To do this we will ask students to 
"think out loud" as they complete the test. The role of the interviewer 
is to prompt the examinee to provide the rationale for both the answer he 
or she chose and for the answers that he or she did not choose. 

Materi a ls Needed 

Instructions to students 

Student form of the test (reusable) 

Separate answer sheet on which students will mark their answers 
One copy of the interviewer form of the test for each examinee 
Tape recorder 

One 90-minute cassette tape for each examinee 
3-4 pencils 

Reliable clock or watch 
Clipboard 
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Overview of Procedure 



The interviewer will introduce the task to the student (see the next 
section} and will distribute testing materials. The student will get the 
student form of the test and will wark his or her answers on the separate 
answer sheet • The interviewer will make sure that students are providing 
the rationale for their choices through probes and will wake notes on the 
interviewer form as they go. The taped version will be used to clarify 
interviewer ootes as needed. The interviewer will also circle any words 
that the student does not understand and provide the meanings for these 
words. Also, if the student does not understand a question, circle it 
and provide a clarification (without giving away the answer). 

Questioning Strategy . Conduct the interview in a non-leading 
fashion. We wish to influence students' thinking as little as possible. 
When the student tells you which answer he or she thinks is correct do 
not let the student know if it is right or wrong. Do not point out flaws 
in the students* reasoning. You will ask questions mainly to lead the 
student to clarify why the student chose the answer he or she did. 

o Ask the rationale for the right answer* Then ask why the 
student thought each of the other choices were wrong • 

o Sometimes the reasons given for choices are not reasons. They 
may repeat the choice, for example. Ideally, we need to know 
the criteria by which a student judged answers, find why the 
choices did or did not satisfy the criteria. 

If the student does give a criterion for choosing and if the 
student makes clear why the choice fulfills that criterion and 
the other choices do not, go on to the next question* 

If the student identifies a criterion but does not state why the 
choice fulfills that criterion ask "What difference does that 
make to your thinking?" 



If the student does not identify a criterion or state why a 
choice fulfills a purpose (e.g., the student repeats the choice 
with little or no elaboration), ask "Could you explain a little 
more what makes you choose one more than the other?" For 
example, in question one the student says — "This is the right 
answer because they need to see what the benefits from the 
project will be and what its gonna do on the environment." Ask 
— "Why is this roost important issue to decide?" Or ask "Could 
you explain a little more what makes you choose this mote than 
the other answers?" 



Interruptions To the Student's Narrative . When students are reading 
items interrupt only to clarify meanings. When students are telling you 
the reasons for their choices and non-choices wait until they are 
finished talking. Then ask for any clarifications. 
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Cautions ^ 

1. Do not rush the interview by beginning to speak iroedia^ely after 
the examinee stops speaking. Wait for a few seconds for the 
examinee to continue. 

2. Do not cut off examinee's reasoning by signalling that enough 
has been said, even though laany examinnes will appear by the 
tone of their voices to seek such signals. 

3. Do not endorse or criticize examinees' fact finding oi reason 
giving. 
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Instructions To Students 



A. Introducing the student to the task 



There is no set script for introducing students to the task, but be 
sure to cover the following points. Keep talking until the student seems 
to be relaxed and ready to begin. 

!• Inform the student of our purpose — to attempt to develop the 
best test we can of people's ability to reason out the answers 
to questions. 



2. Inform the student of his or her role — to give us information 
about how people think when they take our test so that we can 
change the test where changes need to be made. 

3. Inform the student that we are interested in finding out about 
the test and not about the person taking it, so there is no 
reason to feel any stress or pressure. 

4. Inform the student that we want to find out as best we can what 
he or she is thinking while choosing answers to the questions, 
and that to do this we will be asking some questions as the test 
is taken. 

5. Tell the student that we are taping the session so that if the 
notes the interviewer is taking are incomplete we can refer back 
to the tape. 

6. Ask the student the questions on the Interviewer Form Cover 
Sheet — name, major and GPA. 



B. The tape recorder 



1. Label the tape with the student's name and start the recorder. 
C. Beginning the test 



1. Read these instructions to the student: 



■Read each question out loud. Tell me any words or questions 
that you don't unr^erstand. We need to know these so that they 
can be changed. I will tell you the meaning of the word or the 
question. Then pick the answer you feel is best and tell me why 
you chose that answer. Also tell roe why you didn't choose the 
other answers. ■ 



D. After the test 



1. Ask the student the questions on the last page of the 
Interviewer's Form of the Test. 



2. Fill out the postinterview questions on the Interviewer's Form 
cover sheet. 
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CODES USED FOR SCORING STUDENTS* RESPONSES TO HOTS ITEMS: 



0: No reason given/guessing/question too hard/ interviewer led student to 
answer 

1 : Good reason for why R is right 

2: Good reason why a D is right 

3: Good reason why a P is wrong 

4: Good reason why R is wrong 

5: Bad reason wK' R is right 

6: Bad reason why a D is right 

7: Bad reason why a D is wrong 

8: Bad reason why the R is wrong 
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Test Publisher's Panel 



Catherine Ross representing Academic Therap>, Gale H. Roid representing Psychological 
Corporation and Margaret Hill representing SRA 
Moderator: Ken Bumgarner. 



Gale Roid began the presentation by giving an historical perspective to the increased 
demand for teaching and assessing higher order thinking. During the 70s and 80s students 
began showing unexpected lower performances in inferential thinking. An analysis of NAEP 
items also showed less student competence with inferential items. Hr hypothesized that the 
back to basics approach in education may have contributed to the decline. 

Dr. Roid asserted that the MAT 6 is the first battery which tried to separate a HOT 
index from an achievement battery. He feels that the MAT 6 and the SAT 7 Plus are major 
test batteries which measure HOTS across the curriculum. Other tests, such as the Watson 
Glaser, are tests to appraise specific elements of HOTS. 

The test to use should be determined by: 

1. The instructional approach and 

2. The purpose of testing. 

Margaret Hill discussed the new SRA Survey of Basic Skills which will have an 
individual HOTS score and national norms generated from 60 items imbedded within the tfit. 
The test also gives a client the possibilit> of customizing the test with their own items. 

The HOTS questions in the SRA are included in the content areas. Dr. Hill reported 
that, in the reading section, 3/4 of the items deal with analytical or inferential information. 
The Language Arts items address the various tasks required in writing. The math items are 
non-routine items which include extraneous or insufficient information and which ma> require 
estimation. In both Science and Social Science, the students are assumed to have some 
knowledge and HOTS are tested within the content area. All HOTS items are "trying to press 
students at higher levels of thinking." 

Catherine Ross began by discussing the behaviors various thinkers consider as important 
in writing HOTS test items. The views of Lippman, Ennis, Bloom and Costa were briefly 
discussed. 

The Ross Test (designed for grades 4-6) includes items to assess specific skills of HOT 
with 8 test sections. These are tied to Bloom's Taxonomy. 

1. Analogies, 

2. Deductive Reasoning, 

3. Missing Premises, 

4. Abstract Relations, 

5. Sequential Syntheses, 

6. Questioning Strategies, 

7. Analysis of Relevant and Irrelevant information, 

8. Analysis of Attributes (a concept formation activity) 

The Ross test is not intended to be comprehensive, but is planned to be a verbal 
expression of student HOT skill. 
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Ken Bumgarner summarized by saying that the presentations were a reminder that we 
can gain a great deal of knowledge about student HOTS from paper and pencil tests if items are 
creatively written. 
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Higher Order Thinking Skills and Item Banking 



Richard Naccarato and Ray K. Miller 



The first portion of this session was devoted to summarizing the results of an Item Bank 
survey which was conducted by NWREL in 1984. It was pointed out that very few of the banks 
surveyed had categorized their items by cognitive level, in order to allow selecting items which 
measured higher-order thinking skills. Some advantages/disadvantages of categorizing items b> 
certain of these characteristics would be: 

Advantages: 

being able to select items to form a test of primarily **HOTS" knowledge 
equating or adjusting difficulty levels of instruments based upon HOTS skills levels 
Disadvantages : 

HOTS taxonomical classifications for a particular item may spread across numerous 
categories 

the item difficulty levels would become confounded between subject knowledge content 
and thinking skill being tapped. 

Rich Naccarato also described a newer version cf the Item Bank survey which will be 
going out in the near future. 

A major portion of the remaining session time was consumed by the description of 
various taxonomies including Bloom's, Northwest Evaluation Association's, and the Center for 
the Study of Evaluation. Specific subparts of these taxonomies were compared and contrasted. 
The main message(s) behind this comparison was: 

to point out similarities between taxonomies 

to show that none of the taxonomies clearly depicted all higher-order thinking skills 
levels. 

Samples were given of "higher-order" items that exist within a separate bank of 200-300 
items belonging to the NWEA, and there was some discussion of how these would be classified 
within an item bank. Another product of the NWEA was discussed, the upcoming Science Item 
Bank, which will have taxonomical classifications that will be more amenable to selecting HOTS 
items within the science subject area. Participants were later encouraged to distribute the Item 
Bank survey form to anyone whom they knew currently was using a bank, particularly if the 
bank included HOTS items. Numerous handouts were presented, including an item bank surve> 
form which is found in Naccarato (1987) "A Guide to Item Banking in Education, 3rd Edition*'. 
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ITEM BANK DATA SHEET 

Please type or write legibly. This questionnare will be photocopied. 
Item Bank Title {if any) — 



NACCARATO/MILLER 
HANDOUT 

Northwest Regional Educational Laboratory 



For inlcmistioo about the 
bank contact: 



Name. 
Title . 



Organization 

Street _ 

City 

Phono: Area code 



.State. 



.Zip 



NiffT^er . 



Please indicate the characteristics oi thif. item bank by placing an 

1. Which oi the following are available through this 
bank? (check all that apply) 

Q test items 

Q classification of items by content 

□ general objectives or topic statements 

□ item specifications, detailed content descriptions. 

etc. 

□ suggested instructional activities 

n cross references between objectives arvd 
appropriate instructional materials 

□ content review or other validitv information 

□ reliability estimates 
Q p-values 

□ IRT (Ultnl trait) calibrctiont 
Q other item analysis data 
Q technical r.^ports 

2. What is the source of the bank's objectives and 
items? (Check all that apply) 
Q developed by teachers 

□ developed by state or local central office staff 
n developed by test development personnel within 

your organization 

□ developed by an outside organization 
□collectfxi from other sources 

3 What reviews or studies (if any) were perfomied 
' for the items in the bank? (check all that apply) 

□ review to verify appropriate content 

□ content review to match items to objectives 

□ content review to establish oppropriate grade levels 

or age levels 

□ editing for clarity 

□ editing based on reviews by technical personnel 

□ editing based on technical data 

□ review for sex bias 

□ review for cultural end ethnic bias 

n informal pilot testing {infonnal selection of subjects 
^ small numbers of subjects, non-rigorous analysis of 
results) 

rifonnal pilot testing (rigorous sampling and aniysis of 
^ results, large numbers of subjects, standardized 
administration) 



'X" in the appropriate boxes. 
4. Which of these statements are correct lor this 



bank? 



YES 
□ 



CAN BE 
ARRANGED 

□ 



NO 
□ 



□ 
□ 



all or most of the bank 
is available for sale 
ell or most of the bank 
Is available free (or 
fo: cost of reproduction) 
pre-developed tests are 
constructed from the 
item bank 
tests arc constructed 
• based on objectives in 

the item bank selected 
by the user 

□ □ □ items are used to con* 

struct tests based on 
objectives developed 
by the user 

5, What additional services related to thft bank 



printing of test materials 

test scoring services 

development of individual 
student profiles 

development of class and 
school profiles 

training on test administra- 
tion procedures 

training on writing or 
selectirvg objectives and 
item specifications 

training on writing test 
items 

assistance in interpreting 
test data 



can 


be provided? {check 




CAN BE 




YES 


ARRANGED 


NO 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



other 
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O Higher Order Thinking Skills 



I — 



Which of tht Wtewinj «n b« uwd to 
r«tri<vc Items? (ChKk alt that apply) 

□ Uim «ffia»Uy 

□ Utm lyt>« (t.|.. mulUpIi-choici) 

□ cofnilivi l«vi! (•.|., fKitl. Wirinci) 

□ other . — 

□ non« 



NACCARATO/MILLER 
HANDOUT 

K your h^s are rotrievablc by 
cognitive (evel, please indicate from 
where your cognitive categories were 
derived. (If you' would like to attach a 
listing of the taxorK>my you use, we 
would very much appreciate it.) 

D Blo«m*i TMooomy 

Q othti wtlt-known cofnitivt 

Uxonomy, ^utt Htt 



□ mo^ifitd an txitUng Uxonomy, 
^et litt 



Q 4«vf(opod our own coQnittvt 
Uxonomy 



□ olh«r. 



9. in what form are the following available? (Check all that apply) 



Horn ttxts 
Horn graphics 
Kern ttfltistics 



HARD 
COPY 

D 
□ 
□ 



MICRO 
DISK 



□ 
□ 
□ 



MAINFRAME 



NOT 



DISK OR TAPE AVAILABLE 

□ □ 

□ □ 

□ □ 
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10. If compute; software is used to support item banking, please answer the following questions (In each case, 
check alt responses that apply.) 



What function(s) does this 
software perform? 

O Htm manaftrntnt 

Q Utt 4tvdopmint 

Q tcofing 

□ f« porting 

Q ttudint rtcoriHcitpini 

Q crott-rcftrincf to mattriatt 

Q oo-(ini Utt idminittration 

□ othir_ — 



b From where did you obtain this 
software? 

□ adaptid from uittins tprudih^itt. 
databatii, ttc 

□ tf you chiclctd thi abovt box. pttasi 
list Ihi pror^mt you art u»in| 



□ tptcially ^iWffnad for o«r tyitem by 



if your software handles 
item management, indicate 
its capabilities below: 

Q ttortt it<mt 

Q full-tcrctn tditinc 

Oun edd/^tlfitt/chanfi 
Kami 

□ can 8dd/d«)iti/chanst 
Utm clattificationt 

Q will handit uttr history 

□ will handle tttt 
tnatyut 



□ purchattd from tttt of rtttarch 
orianiution 



□ olhtr_ 



erJc 
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d. If your softwart handles 
t€St dcvslopment, MicdU 
its capabilities Mow: 

□ oo-Unt {cnealion of 
Uttt 

can uftc vtriMt crittda 
□ fof MlKtins Utmt 

_^ Mn Hd/dtl#lt/chdn|t 
n H«m$ that «rt 
com^tor Mifcttd 

•utomatlutly itortt 
U antwtr ktyt 

□ has social print 
featurat 

□ win print multiptt 
focmt 

O othtf_ 



If your eoftwire handles 
scoring, piaase indicate its 
edibilities b«low: 

0 marii MAM rudof 

□ tu^ttst and tfrta! 
KOftt 

O ^Jocthrc matt^ 
Q Ham fUtistict 

□ Ust sUtiitici 

□ ethtr 
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HANDOUT 
f ff your software handles 
reporting, please indicate its 
capabilities below 

SuMtst ^ofittt — 
U ttudtnt 

□ lubtctt rofilti — cJaii 

£2 Mmmory rcporti 

aoti r«f«ftnct to 
Q mattnali/mtthodt 



Q othtf. 



If your software handles 
student recordkeeping, please 
indicate its capabilities 
b^low: 

Q objtctivt matttrtd 

Q total Kortt ovar timt 

_ fradtbook or itriet of 
[j icortt 



if your software has 
graphics capabilities, please 
indicate these below: 

□ can fanoratt moit 
(raphici 

can attach axttmal 
[jl ffraphict to ittm taxt 

can gtntratt iraphici 
n and merft thtm in 
with Ittm ttxt 

□ can ^oduct tht antirt 
tttt 



i. What type of computer do 
you use'^ (If microi please 
indicate XT. AT. 286. etc.) 



What is the approximate 
cost of your testing 
software'^ 



]. What is the name of your item 
banking/test scoring software 
package^ 



k. Is your software:? 



avallablt on uchangt 
Q asraamtnt 

_ availabit for purchata 

LJ from vandor 

availabit for porchait 
□ from you 

Q not availabit for othtrt 



m. Technical assistance in 
running the software is 



available from ui 



avaHabtt through 
□ vaodor 



available 



11 Please provide any descriptive comments or explanatory information about your item bank and/or testing system 
m this space or on a separate sheet of paper. We are especially interested in items measurmg higher order 
thinking skills, and in computerized systems 
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Fro^s Millman & Arter, 1984 

Questions to be Answered in Designing Itexa Banking Systeas 

I. ITEMS 

A, Acquisition and Developiaent 

!• Develop/use your own itea collection or use collections of 
others? 

a. If develop your own item collect ion , what develc^iaent 
procedures will be followed? 

b. If uae collections of others, will the iteas be leased or 
purchased, and is the classification scheiae sufficiently 
documented and the item format specifications sufficiently 
compatible for easy transfer and use? 

2. what types of "items" will be permitted? 

a. Will open-ended (constructed response) items, opinion 
questions, instructional objectives, or descriptions of 
performance tasks be included in the bank? 

b. Will all the items be made to fit a common format (e.g., 
all multiple-choice with <^tions a, b, £, and d ? 

c. Must the items be calibrated, validated, or oti^erwise 
carry additional information? 

3. What will be the size of the item collection? 

a. How many items per objective/subtc^ic (collection depth)? 

b. How many different topics (collection breadth)? 

4. What review, tryout and editing procedures will be used? 

a. Who will perform the leview/editing? 

b. Will there be a field tryout, and if so, what sb^tistics 
will be gathered, and what criteria will be used for 
inclusion into the bank? 

B, Classification 

1. How will the subject matter classifications be conducted? 

a. Will the classification by subject natter jse fixed 
categories, keywords, or some combination of the two? 
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b. Who will be responsible £or preparing the taxonomy? 

c« How detailed will the taxoncQy be? Kill it be 
hierarchically or nonhierarchically aranged? 

d, Hho will assign classification indices to each item, «md 
how will this assignment be verified? 

2* What other assigned inforaation about the itens will be stored 
in the item bank? (See the attached list for potential 
attributes.) 

3, What measured information about the items will be stored in the 
bank? (See the Appendix B list for potential seasures.) How 
will the item measures be calculated?* 

C. Management 

1« Hill provision be made for updating the classification scheme 
and items? If so: 

a. Who will be permitted to make additions, deletions r and 
revisions? 

b. What review procedures will be followed? 

c. How will the changes he disseminated? 

d. How will duplicate (or near duplicate) items be detected 
and eliminated? 

e. When will a revision of an item be trivial enough that 
item statistics from a previous version can be aggregated 
with revisions froca the current version? 

f« Wjll item statistics be stored itcssx eoch use, last use, or 
^iggregated across uses? 

2. How will items that require pictures, graphs, special 
characters, or other types of enhanced printing be handled? 

3. How will items that must accoc^any other items, such as a 
series of questions about the same reading passage, be bandied? 



*This question is the subject of considerable controversy and discussion 
in the technical measurement literature. For example, to obtain a latent 
trait difficulty parameter, concern has been expressed about sanple size, 
calibration procedure (Rasch, 3-parameter) , linking models (major axis, least 
squares, maximum likelihood) • and number of items cozsmcn to the equating forms* 
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II. TESTS 



A. Asseably 



1. Must the test constructor specify the specific items to appear 
on the test or will the items be selected by the computer? 



2. If the items are selected by the cog:?>uter: 

a. How will one itea out of several that catches the search 
specification be selected (randomly, ti©e since last 
usage, frequency of previous use)? 



b. What happens if no item meets the search specifications? 

c. Will a test constructor have the option to reject a 
selected item, ^nd if so, what will be the mechanism for 
doing so? 

d. What precautions will be taken to insure that examiners 
who are tested more than once do not receive the same 
items? 



3. What item or test parameters can be specified for test assembly 
(item foraat restrictions, limits on difficulty levels, 
expected score distribution, expected test reliability, etc.)? 

4. What asseably procedures will be available (options to 
multiple-choice items placed in random order, the test items 
placed in rjmdom order, different items on each test)? 

5. Will the system print tests or just specify which items to 

use? If the forraer, how will the tests be printed or ^ 
duplicated and wliere will the answers be displayed? 

B. Administration, Scoring and Reporting 

1. Will the system be capable of on-line test administration? 

If so: © 



a. How will access be managed? 

b. Will test administration be adaptive, and if so,^ using 
what procedure? 

2. Will the system provide for test scoring? If so: 

a. What scoring formula will be used (rights only, correction 
for guessing, partial credit for some answers, weighting 
by discrimination values)? 
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b« How vill Constructed responses be evaluated (off-line by 
the instructor, on-line/off-line by cxaniners collaring 
their answers to a key, orv-line by cocnputer with/without 
esploying a spelling algoritha)? 

3. Will the systen provide for teat reporting? If so: 

a« Mhat records will be Kept (the tests themselves, 

individual student itea responses/ individual student test 
scores, school or other group scores) and for how long? 
Will new scores for individuals and groups supplement or 
replace old scores. 

b. What reporting options (content/ format) will be available? 

c* To \th(xs^ will the reports be sent? 

C. Evaluation 

1. WiJ^ reliability and validity data be collected? If so, what 
data will be collected by whom, and how will they be used? 

2« Will norms be made available and, if so, based on what 
norm-*referenced measures? 

III. SYSTEM 

A. Acquisition and Development 

1. Who will be responsible for acquisition/development, given what 
resources, amd operating under what constraints? 

2. Will the system be made transportable to others? What levels 
and what degree of docuzaentation will be available? 



B. Software/Hardware Features 

1. What aspects of the system will be con^uter assisted? 
a. 



b. 



Where will the iteas be stored (computer, paper, card 
file)? 

Will requests be filled using a batch, on-line, or manual 
eode? 



2. 



3. 



Will items be stored as on large collection or will separate 
files be maintained for each user? 

Bow will the item banking system be constructed (from scratch; 
by piecing together word processing, data-base management, and 
other general purpose programs; by adopting existing item 
banking systems)? 
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4, What specific equipment will be needed (for storage, retrieval, 
interactions with the ayoteor etc#)? 

5, Bow user and aaintenance friendly will the eguipnent and 
support prograas be? 

€• Who will be responsible for equipaent eaintenance? 

C. Monitoring and Training 

1* What syatea features will be aonitored (nuaber of iteas per 
classification category, usage by user group, number of 
revisions until a user is satisfied, distribution of test 
lengths or other test characteristics, etc.) 

2. Who will Bonitor the system, train users, and give support 
(initially, ongoing)? 

3. How will information about changes in system procedures be 
disseminated? 

D. Access and Security 

1. Who will have access to the iteiac and other information in the 
bank (authors/owners, teachers, students)? Who can request 
tests? 

2. Will users have direct access to the systea or oust they go 
through an interiaediary? 

3. What procedures uill be followed to secure the contents of the 
iteo bank (if they are to be secure)? 

4* Where will the contents of the item bank be housed (centrally 
or will each user also have a copy)? 

5. Who will have access to score reports? 

IV, USE AND ACCEPTANCE 

A* General 

1. Who decides to what usee the item bank will be put? And will 
these uses be the ones that the test use " need and want? 

2. Who will develo|) the tests and who will be allowed to use the 
system? Will tiiese people be acceptable to the exeuuinees and 
recipients of the test information? 

3. Will the system be able to handle the expected demand for use? 
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4, Mill the output of the systeo likely to be used and used as 
intended? 

5. How will user acceptance and iteo bank credibility be enhanced? 
Instructional Iinprovement , If this is an intended use: 

1. Will the it^ bank be part of a larger instructional/decision- 
making system? 

2. Which textbooksr curriculua guidelines, and other materials,, if 
any, will be keyed to the bank's items? Who will make that 
decision and how will the assignments be validated? 

3. Will items be available for drill and practice as well as for 
testing? 

4. Will information be available to users that will assist in the 
diagnosis of educational needs? 

Adaptive Testing , If this is an option: 

1. How will the scheduling of the test administrations take place? 

2. How will the items be selected to insure testing efficiency yet 
maintain content representation and a^oid duplication between 
successive test administrations? 

3. What criteria will be used to terminate testing? 

4. What scoring procedures will be followed? 

Certification of Cotg^etence , If this Lz an intended use: 

1. Will the item bank contain measures that cover all the 

important cosz^nent skills of the coo^tence being assessed? 

2* How many atteE5>ts at passing the test will be allowed; when? 
Bow will these attempts be laonitored? 

Program/Curriculum Evaluation . If this is an intended use: 

1. Will it be possible to ic^lement the system so as to provide 
reliable measures of student achieveaient in a large number of 
specific performance areas? 

2. Will the item bank contain measures that co^'er all the 
important stated objectives of the curriculum? That go beyond 
the stated objectives of the curriculum? 

3. Will the item bank yield commensurable data that permit valid 
ccHoparision^ over time? 
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F. Testing and Reporting ReguireEents Imposed by External Agencies . If 
meting these requirements is an intended use: 

!• Will the system be able to handle requirements for progran 

evaluation (e.g.r Chapter 1), student selection into specially 
funded programs, assessing educational needs, and reporting? 

2. Will the system be able to acco^date lainor mdifications in 
the testing and reporting requirc^nts? 

V. COSTS 

A» Cost Feasibility 

1. What are the (fixed, variable) costs (financial, time, space, 
equip^nt and supplies) to create and support the system? 

2. Are these costs affordable? 
B. Cost CcHBparisions 

1. How do the item banking system costs compare to the present or 
other testing systems that achieve the same goals? 

2. Do any expanded capabilities justify the extra cost? Are any 
restricted capabilities balanced by cost savings? 
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ITEM RETRIEVAL CHARACTERISTICS 

1. Item Difficulty (classical or otherwise) 

2. Question Format (M-C, T-F, etc.) 
3c Reference to Curriculum 

4. Readibility Level 

5. Item Identification Number 

6. Directions Required (Oral, etc.) 

7. Cognitive Level (Recall, Inference, Application, 

Problem-Solving) 

OTHER POSSIBLE CHARACTERISTICS 

8. Source of the Item 

9. Cross-referencing to Other Goals 

10. Date of last use 

11. I.D. of Item Writer/Editor 

12. Security Level Code 

13. Recommended Grade Level 

14. Content Key Words 
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RESOUffCES FOR INFORMATION ON ITEM BANKING 



Arter, J. A. & Estes, G. D. (Nov., 1985). Item banking for local 
test development: Practitioner handbook . Pub 1 ished by the 
Northwest Regional Educational Laboratory, Portland, OR. 

Deck, D., Nickel, P. &c Estes, G. (Nov., 1985). Reviews of 

microcomputer item banking software. Published by the ^ 
Northwest Regional Educational Laboratory, Portland, OR. 

Educational Testing Service (198^). Item pools. A publication of 
the ETS Test Collection, Educational Testing Service, 
Princeton, N J • 

9 

Estes, 6. D. (Ed.) (Nov., 1985). Examples of item banks to 
support local test development: Two case studies with 
reactions. Published by the Northwest Regional Educational 
Laboratory, Portland, OR. 

Estes, 6.D. &t Arter, J. A, (198^). A guide to item banking in ® 
education (second edit ion) : Item bank data sheets. Published 
by the Nor thwest Reg ional Educat ional Laboratory , Por t land , 
OR. 

Millman, J» & Arter, J. A. (Winter, 198^). Issues in item 

banking. Journal of Educational Measurement, 21, pp. 315- @ 
330. 
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Cross References for three HOTS SCHEMES for Cognition 



No. CSE-UCLA 



Mnemonic strategies 



Cognitive strategies 



Cognitive styles 



Communication skills 



Translation Si Interpretation of ld«as 



Comparing 



Sequencing 



Classifying 



Categorizing 



Inventing-ideating 



Planning 



Problem redefinition 



Transfer of approach to new problems 



Creating Hypothesis 



Inferring from data 



Deductive Reasoning 



^ Evaluating according to logical criteria 



Specifying judgemental criteria 



Sensitivity to missing. Irrelevant or misleading InfrnnatlcTi 



Discriminating statements of valuo and fact 



Creating situations to provide data for evaluation 



Bulldir^ theories or models 



, j^'iluing (internalizing decisions /plans). 

ERJC I7t) 



No. 



P-37 



P-34 



NWEA Uses 
Tricounty Course Goal Knowledge/Process 
Classifications 



Vocalizing, moving, gesturing, touching, speaking, 
wfitinK. med ia, dramatizing, etc. 



Summarizing, Abstracting 

Classifying, categorizing, grouping, selecting 
accordi ng to criteria 



Ordering, sequencing 

Classifying, categorizing, grouping, selecting 



P-33 



P-67 



P-34-36 



P-54 



P-65 



P-52 



P-42 



P-43 



P-22 



P-47 



P-22 

T=rr 



according t o criteria 
Same as above 



Creating cn basis of knowledge and process 



P 6i-6J Ordering, sequencing, arranging, 
transform^ estimste, summarize, decide, solve, adapt 



Revising Hypothesis 



Restructuring behavior, adapting, modifying 



Formulating hypothesis 



Inferring, interpolating, ext^apolatttig 



Generalizing 



Evaluating logical consistency and accuracy 



P-53 



P-51 



P-64 



Testing against standards or criteria 

Evaluating authoritativeness, logical consistency, 
relevance, and adequacy 



No. 



220 



420 



1.30 



1.30 



5.00 



5.20 



5.30 



3.00 



5.30 



510 



6.00 



Testing against standards,/crtteria 
Clygsifyins, according to criteria 

Testing hypothesis 



Theorizing, Predicting 



Restructuring values (adapting, modifying) 



3.00 
6>00 



6.00 



6.00 



T55 



Bloom's Taxonomy 



Comprehension-Translatton 
Efttrapo^atten 



Knowing waya & nnoant to 
sa> with ot >eciricfl 



96e above 

Analysis of c^eniialtwal p^inc^W 



Knowledge of universafs 
and abstractions in a fieM 



See abova 



Synthesis 



Production of a plan 



Derivation of a set of 
at>stract relations 



Application 



Oertvatk>n of a sat of 

abfltfat;t j 



Production of unique 

f OTmunic^tbna 
valuation using internal/ 
external €v}<fence 



Analysis 
Evaluotiyn 
Application 



Eveluatipn ^ 

Evalulatton in terms of intefnay 



external evidence 



Same aa abova 

Production of a plan or 
proposed set of operat ions 
Derivation of a set of 
abstract relations 



B > 
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NW£A Reading 8arU< Index 

PASSAGES 

5.2 Reading 

5.2. 1 Readinass 

5.2.2 Knowledge of Ccmventions 
5.2.5 Word Recognition, Word Meaning 

5.2.3.5 Strtxitural Analysis 

5.2.3.4 Content Clues 

5.2.5.5 Homonyms, Synonyms, Antonyms, Acronyms, Malapropisms 

5.2.4 Literal Reading Comprehension 

5.2.4.1 OetaUs 

5.2.4.2 Main Ideas 

5.2.4.3 Seqimnce 

5.2.4.4 Classificatioii 

5.2.4.5 DIrecUons 

5.2.5 Interpretive Comprehension 
5.2.5. j Inference 
5J2.5.2 Analysis 

5„2.5.3 Association, Cause-Effect 
5.2.5.4 Comparison, Contrast, Analogy 
5.2.5.3 Summary, Synthesis 

5.2.5.6 Generalization 

5.2.5.7 Prediction 

5.2.6 Critical/Evaluative Comprehension 

5.2.6.1 Fact-Opinion 

5.2.6.2 Persuasion 

5.2.6.3 Intemal/Extemal Validation 

5.2.6.4 Assumptions, Author's Attitudes 

5.2.7 Applications 
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MATHEMATICS NACCAR^^TO/MILLER 

HANDOUT 

^Knowledge Categories • 

Gl Principles and Laws 

G2 Simple Generalisations 

Kl Convetttions: Names and Nomenclature 

K2 Conventions: Symbols, Rules, Standardized Processes, 

Definitions 

K3 Properties, Pcirts, Characteristics, Features, Elements, 
Dimensions 

K4 Trends and Sequences 

'/ - K5 Similarities and Differences, Discriminations, 
Classifications 

K6 Contexts, Locations, and Orientations 

^ K7 Operations, Methods of Dealing with. Functions 

K8 Cause and Effect Relationships (Costs and Benefits) 

K9 Ci'iteria or Standards 

KIO Non Cause -Effect Relationships 

-Inquiry-Problem Solving Processes- 

Input Acquiring Information 

Pll Viewing 

P12 Hearing 

P13 Feeling (tactile) 

P14 Smelling 

P15 Tasting 

P16 Using sense extenders 

P17 Using internal' sensors of emotion 

Input Insuring Validity and Adequacy 

Verification 

P21 Evaluating authori tativeness of sources 
P22 Evaluating logical consistency and accuracy 
P23 Evaluating relevance to desired learning purposes 
P24 Evaluating adequacy for acting or deciding 
(comprehensiveness and depth) 
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Preprocessing Organising Information 

P31 Labeling, naming, numbering, coding 

P32 Recording, listing ^ 

P33 Classifying, categorizing, grouping, selecting, 

according to criteria 

P3A Ordering, sequencing 

P35 Manipulating, arranging, transforming, computing 

• P36 Estimating 

P37 Sunmarizing, abstracting ^ 

Processing I Interpreting Information (drawing meaning from data) 

PAl Decoding verbal and non-verbal symbols 

(reading and literal translating) 

P42 Inferring, interpolating, extrapolating ,y 

P43 Analyzing 

P44 Associating, relating, equating 

P45 Comparing, contrasting, discrir-inating 

- P46 Synthesizing 

P47 Testing against standards or criteria 

P48 Generaliz ing • 

Processing II Using Information to Produce New Information 

P51 Theorizing, predicting 

P52 Formulating hypotheses 

P53 Testing hypotheses ® 

P54 Revising hypotheses 

Output I Acting on the Basis of Information 

P61 Reacting ^ 

P62 Making decision ^ 

?63 Solving problems 

P64 Restructuring values (adapting, mod. -ying) 

P65 Restructuring behavior (adapting, modifying) 

P&6 Encoding verbal and nonverbal symbols prior 

to communication ^ 

P67 Creating on the basis of knowledge and process 

Ou tput II Communicating Information 

P71 Vocalizing (nonverbal) 

P72 Gesturing, moving 

P73 Touching 

P74 Speaking 

P75 Writing 

P76 Using art media (painting, drawing, 

sculpting, constructing, etc.) 

P77 Dramatizing 

P78 Singing, playing instrvm)ents 

P79 Dancing 
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Alternative Formats For Assessing Higher Order Thinking Skills 

Edys S. Quellmalz 



The purpose of Dr. Quellmalz's presentation was to provide the audience with a variety 
of alternative formats for assessing HOTS. She began by describing categories of HOTS skills 
and processes that she has found useful when attempting to assist districts to examine HOTS. 
This list is provided on page 1 of her handouts. 

a Analyze Identify components of things; what something is made up of. 

a Compare -- Compare properties, events, features of two or more things, 

a Infer/Interpret Draw conclusions. 

a Evaluate — Apply criteria to make and defend solutions or judgements. 

Dr. Quellmalz finds that these thinking processes cut across all content areas and that the 
terminology used by philosophers to describe "critical thinking", psychologists to describe 
^'problem solving", etc. both really describe these same basic set of processes. These points are 
illustrated by pages 2 and 3 of her handouts. Figure J shows how terminologv across disciplines 
can be aligned; Figure 3 shows how the HOTS processes described above apply to various 
content areas. 

Before developing assessment questions that measure these processes, one >nust make 
several decisions. 



1 V-Ml skills be measured in isolation or will the integrated use of skills be emphasized? 
This point refers to the issue of reductionism discussed in the keynote address. 

2 How will mei3cognitivc skills be handled? Metacognitive skills are .isted on page 1 of 
Dr. Ouellnvalr': hw"^ndout. 

3 Wi!' the content of the questions deal with significant issues, recurrent issues or novel 
situ'-tions with wliich the student mu5't deal? 

4. H:)w will prior knowledge be handled? Will you deal with situations that should be 
within the persona! experience of students; information that students have been taught 
at school; oi will all information be provided in the test itself? 

5. How will questions be presented? This includes presentation mode such as written, 
visual, or demonstration; the amount of information in the question that must be 
processed bj :he student; whether information will be provided on the criteria by 
which a response will be evaluated; and response format such as written, oral, 
performance, group, individual, 

6. Will the test be timed? 

7. How will the assessment be tied into instruction. Dr. Quellmalz provided an example 
(see page 4 of her handout) of how a set of criteria and a listing of what needs to be 
included when one uses a HOTS process can be used to both guide instruction and 
evaluate student responses. 
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Dr. Quellmalz then proceeded to show a variety of questions that she has helped 
developed that attempt to assess the various HOTS skills and processes in a number of content 
areas. The following represents a few of those presented: 



1. Essay. An example would be "Compare the Russian and U.S. versions of the Russian 
revolution.*" For this type of question, Dr. Quellmalz includes extra informatijn for 
the student passages from a Russian and U.S. textbook, what should be covered 
(such as "what would they both agree on, disagree on and why"), and .le 
information about how it will be scored (such as "be sure to support your arguments 
with examples from the text" or "be sure to explain your position, describe your 
criteria for deciding on a position, and provide examples which support your 
opinion"). 

2. Partial Essay. In this question type, part of the essay is already written. The student 
is asked to complete it by adding supporting examples, etc. This format might be 
useful for students just learning how to develop arguments. 

3. Combination multiple-choice, short answer and essay. Combining the question types 
covering a presented situation might allow for a more complete measures. Multiple- 
choice questions could cover finding information and other HOTS skills measurable in 
multiple-choice format short answer questions and essays make students explain their 
reasoning. Also increasing the number of questions asked about a situation enables 
one to provide a more complex situation. 



In preparation for the next session that Dr. Quellmalz presented jointly with Dr. Rick 
Stiggins, she asked participants to classify the questions on pages 5 and 6 of her handout as to 
what level of thinking skill each one taoped. 
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FIGURE 2 
HIGHER ORDER THINKING 



QUELLI'Jvi: 



*( 



Students engage in proposeful, extended lines of thought where they: 

- Identify the task of problem type 

- Define and clarify essential elements and terms 

- Judge and connect relevant information 

- Evaluate and adequacy of information and procedures for drawing conclusions 
and/or solving problems 

In addition, students will become self-conscious about their thinking and 
develop their self-monitoring problem solving strategies. 



COMMONLY SPECIFIED 
HIGHER ORDER REASONING PROCESSES 



COGNITIVE 
Analyze 
Compare 
Infer/Interpret 
Evaluate 

METACOGNITIVE 
Plan 
Monitor 
Review/Rovise 
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FIGURE 1 

RELATIONSHIP AMONG REASONING SKILLS PROPOSED IN PSYCHOLOGY AND PHILOSOPHY 



CRITICAL THINKING SKILLS 
(Philosophy) 



PROBLEM SOLVING 
STRATEGIES 
(Psychology) 



PROBABLE DOMINANT 
COGNITIVE PROCESSES 
(Psychology) 



Clarification 

- Identify or formulate 
a question 

- Analyze majt; components 

- Define important terms 



- Identify the problem 

- Identify essential 
elements and terms 



Analogical 
analysis 
comparison 



Judge credibility of support, 
the source, observations 



Inference 

- deduction 

- induction 

- value judgments 

- fallacies 

Use criteria to judge 
adequacy of solutions 



Identify appropriate 
information, content 
and procedural schemata 



Connect and use 
information to solve 
the problem 



Evaluate success of 
the solution 



Analogical 
analysis 
comparison 
evaluate components 

Inferential 
infer/ interpret re- 
lationships among 
components 



Evaluative effective- 
ness of specific and 
general strategies 
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Figure 3. Examples of Higher-Order Reasoning Skills in 
Three Subject Domains 



Science 



Social Science 



Literature 



Analyze Identify the com- 
ponents of process 
and the feanires of 
animate and inani- 
xnatc objects 



Analyze compo- 
nents or elements 
of an event 



Identify compo- 
nents of liieraiy, 
expository, and 
persuasive dis- 
course 



Compare 



Compare the prop- 
erties of objects or 
events 



Compare causes 
and effects of j>cpa- 
rate events; com- 
pare social, politi- 
cal, ccononuc, cul- 
tural, and geo- 
graphic features 



Compare meanings, 
themes, plots, char- 
acten, settings, and 
reasons 



( 



Infer 



Draw conclusions; 
make predictions; 
pose hypotheses, 
tests, and explana- 
tions 



Predict, hypothe- 
size, and conclude 



Infer characters' 
motivation; infer 
cause and effect 



Evaluate Evaluate soundness 
and significance of 
findings 



Evaluate credibility 
of arguments, deci- 
sions, and reports; 
evaluate signifi- 
cance 



Evaluate form, be- 
lievability, signifi- 
cance, complete- 
ness, and clarity 
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HIGHER ORDER THINKING (H.OJ-) PROJECT 
THINKING STRATEGIES: COMPARE 

1. Identify the objects, events, ideas to be compared 

I. Identify relevant examples/attributes that are similar or different to 
compare 

3. Identify/list categories of attributes that are similar or different 

4. Explain the reasons or function of the similarities and/or differences 
(so what?) 



EVALUATIVE CRITERIA FOR STUDENT RESPONSES 
Names objects/events/idtfts to be compared 



1. Identifies categories for comparison 

- are they distinctive? 

- is there a sufficient number? 

3. Appropriately identifies and explains how the 
objects/events/ideas compare and/or differ 

on attributes within each category 

- are the comparisons accurate? 

• are the^ balanced i.e. describe differences? 

• how thorough are they? 

4. Presents comparisons in a logical sequence 



5. Explains the significance/function of the 
similarities and/or differences 

- how clearly? 

- how appropriately? 

- how extensively? 



1 


2 


3 


no 


implies 


names 


1 


2 




no 


few 


some 



1 

little 
or none 



confused 



1 

not at 
all 



2 

few 



2 

some 
off 

2 

some- 
what 



3 

some 



3 

BK>stly 



3 

some 
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Final Step; A Progress Checic 

In the space provided next to each exercise, enter the letter that 
represents t^^ thinking sKill category reflected in the item (See Appendix B 
for answers} : 

R » Eecall 
A « Analysis 
C « Cosc^arison 
Z B Inference 
B » Svaluation 

^^^^^^^ 1. What are 3 functions of the liver? 

2. Let's brainstorm what woulJI happen if the sun did not cosae 

up tosaorrow. 

3. Define the word ©itasis. 

4. Which of the following Qenus is the best? Why? 

5. Which aenu provides Dore ccs^lete protein? 

6. Should the use of coaputers bo abolished in the classroom? 
Why or why not? 

^^^^^^^^^^ 7. Who is the author of where the Sidewalk Ends? 

8. If wo mix these chemicals together, what do you suppose 
will happen? 

9. Look at the char^* showing the number of meals Americans 
have eaten away jcro© home in the last three years. How 
have eatir^ habits changed? 

10. What are three purposes of an unmanned space flight to 

Jupiter? 

^^^^^^^^^^^ 11. What are the functions of our eyelashes? 

12* Which do you think will have greater isspact on your life, 
the invention of the computer or our ability to travel in 
space? Why? 

13. If you were going outside and it was snowing quite hard, 
which of the following would you need from your closet? 

a) Your umbrella 
^ b) A light weight jacket 

c) Your warm boots 
d) Your sandals 
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14. 

15. 

16. 

17. 
18. 

19. 

20. 



You hate rain, but know it ifi necessary. What are three 
purposes it serves? 

In the Northwest it rains and .snows a lot. Which is more 
vital for the necessary supply of water for suaafier use? 

What are so©a jobs a migrant worker eight perform in 
jotting a crop of lettuce to earket? 



Haiku is a form of 



took at the three paintings. Which makes the sost use of 
vivid colors? 

Sirppose w® had not dropped the bosts on Hiroshiea and 
Nagasaki f how else might we have defeated the Japanese? 

Which is a better snack for you, a fresh peach or a dish of 
f rosen peach yogurt? Why? 
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Classroom Assessment of Higher Order Thinking Skills 
Richard Stiggins and Edys S. Quellmalz 



Rick Stiggins and Edys Quellmalz teamed up to provide practical strategies for 
teaching teachers to measure higher order thinking skills in the classroom on a day-to- 
day basis. After reviewing a wide variety of reasons why each teacher must be skilled 
in this kind of assessment, the presenters introduced the Assessment Planning Chart, an 
easy-to-use planning device designed to raise teachers' level of consciousness about ' 
thinking skills and their measurement. The chart was taken from ''Measuring Thinking 
Skills in the Classroom** by R. Stiggins, E. Rudel and E. Quellmalz. 

Participants went through a series of steps in learning how to use the chart. 
Essentially, the chart crosses five levels of thinking skills (recall, analysis, comparison, 
inference, and evaluation) with three types of assessment (oral questions, test items, and 
performance assessments) to create five-by-three, fifteen cell matrices. Within each cell, 
the user writes an assessment probe tapping the appropriate level of thinking. 

As a first step in the proct:s of learning to use the chart, participants reviewed a 
completed chart, studying each cell entry to discern how and why it reflected a 
particular level of thinking. In step two, they filled in empty cells in a partially 
completed ch-^rt. In step three, they used a variety of prespecified question forms and 
formats to complete a chart on a topic of special interest to them. And finally, using a 
^'ariety of other instructional aids, each participant prepared a complete chart on their 
own. 

As a conclusion to the session, the participants brainstormed various ways to 
integrate this kind of assessment into actual classroom contexts. The presenters initiated 
the list of ideas based on their experience, and the participating teachers and 
administrators added to the extensive list of suggestions. 
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Table 1 

<^UMMARY OF THINKING SKILLS 



Level 



Recall 



Analysts 



Compansor, 



Infe'exe 



Evaiualion 



Definition 



Most tasks require that students recognize or remember 
key facts, definitions, concepts, rules, and principles Re- 
call questions require students to repeat verbatim or to 
paraphrase given information. To recall information, stu- 
dents need most often to rehearse or practice it, and then 
to associate it with other, related concepts The Bloom 
taxonomy levels of knowledge and comprehension are 
subsumed here, since verbatim repetition and translation 
into the student's own words represent acceptable evi- 
dence of learning and understanding 

In this operation, students divide a whole into component 
elements Generally the part/whole relationships and the 
cause/effect relationships that characterize knowledge 
within subject domains are essential components of mo'e 
complex tasks. The components can be the distinctive 
characteristics of objects or ideas, or the basic actions of 
procedures or events This definition of analysis is the 
same as that in the Bloom taxonomy. 

These tasks require students to recognize or explain simi- 
larities and differences Simple comparisons require at- 
tention to one or a few very obvious attributes or compo- 
nent processes, while complex comparisons require 
identification of and differentiation among many attributes 
or component actions This category relates to some of 
the skills in the Bloom level of analysis Tho separate 
coioparison category emphasizes the distinct information 
processing required when students go beyond breaking 
the whole into parts in order to compare similarities and 
differences 

Both deductive and inductive reasoning fall in this cate- 
gory In deductive tasks, students are aiven a generaiiZc- 
tion and are required to recognize or explain the evi- 
dence that relates to it Applications of rules and 
*'if-then" relationships require inference In Inductive 
tasks, students are given tf;3 evidence or details and are 
required to come up with the generalization Hypothesiz- 
ing, predicting, concluding, and synthesizing all require 
students to relate and integrate information. Inductive and 
deductive reasoning relate to the Bloom levels of applica- 
tion and synthesis Application of a rule is one kind of de- 
ductive reasoning, synthesis, putting parts together to 
form a generalization, occurs in both inductive and de- 
ductive reasoning 

These tasks require students to judge quality, credibility, 
worth, or practicality General, ve expect students to use 
established criteria and explain how these criteria are or 
are not met The criteria might ^^e established rules of 
evidence. logic« or shared values Bloom's levels of syn- 
thesis and evaluation are involved in this category To 
evaluate students must assemble and exclam the inte*- 
relationship of evidence and reasons in support of thei^ 
conclusion (synthesis) Explanatson of crrteria for reaching 
a conclusion is unique to evaluative reasoning 



Relation to 
Bloom Taxonomy 

Recall 

Comprehension 



Analysis 



Analysis 



Application 
Synthesis 



Synthesis 
Evaluation 



ASSEvSSMENT PLANNING CHART 



GRADE LEVEL 



Junior High 



SUBJECT 



Social Studies 



TOPIC 



Electoral College 





ORAL 


TEST 




RECALL 


What is the electoral college ' 


As a member of the electoral college, you must 
vote: 

a. According to your own judgment 

b. As your constituency voted 

c. As the party tells you 

d Only if you wish to do so. 


Assume you*re a U.S. .senator. Propose a con- 
stitutional amendment that would make the pop- 
ular vote the sole criterion for electing a presi- 
dent. Your amendment would do away with the 
electoral college. Prepare a speech to Congress 
defending your amendment. Be sure to— 
a. Analyze all elements of the issue. 


ANALYSIS 


How docs the electoral college work? 


Analyze the steps in the presidential election 
process, showing where the electoral college 
comes into play. 


b. Compare elections with and without the 
college, 

c. Show how the voters are likely to react, and 

d. Slate and defend your values. 

(Or conduct a simulated debate on the Senate 


COMPARISON 


How do the social conditions that existed when 
the electoral college was formed differ from 
conditions now? 


V^hat is meant by the election theme **one pcr- 
.son. one vote/* and how docs that relate to the 
electoral college? 


floor.) 


INFERENCE 


If you were a presidential candidate elected by 
popular vote, could you still lose the election? 
How? 


In which stale is the electorate likely to oppose 
the use of the electoral college: 

a. California 

b. Illinois 

c. Etc. 




EVALUATION 


Should the electoral college be abolished? Why 
or why not? 


Which of the following is the best reason for 
maintaining the electoral college? 

a. Tradition. 

b. Fairness to large states. 

c. Etc. 
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GRADE LEVEL 



ASSESSMENT PLANNING CHART 

High Schtx)! ^>ocial Studies 



lUBJECT 



TOPIC 



ORAL 



TRST 



Branches of Government 



PERFORMANCE 



RECALL 


Who's in charge of ihc executive branch of 
government? The judicial branch.' The 
icgislature? 




ANALYSIS 




What can the executive branch do about an un- 
fair law** 


COMPARISON 


In America we have Congress. In Canada their 
legiOaturc is called Parliament. How do they 
differ in stnicturc? 
OR 

Compare the Mructurc of the federal ?.overn- 
mcnt with that of your state government. 




INFERENCE 




You*ve decided motorcycles cause tw) many ac- 
cidents. You*d like them banned from federally 
funded highways. To which government branch 
would you first appeal? 
a. lixccutivc b. Judicial c. Legislative 


EVALUATION 


Whieh branch of the government is the most 
important? Why? 





Set the class up as a democracy with three 
branches of government. Go through the actual 
process of passing a law. Determine if students 
know and can carry out each functit)n Stnve to 
demonstrate the interactions among branches. 
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XSSESSMBHT PLANHXNG CHART 
8UBJ6CT GanTlc Chart 



TOPIC 



ANALYSIS 



COHPARISOda 



INFBRSS^CB 



OHAL 



Wiat is 



7 



EVALUATION 



to 



(Quit do you think tiould hApcxm 
if ? 



happttfuit do 



TEST 



PKRFORKANCB 



la this • MtiofACtoty aolution to thin 

problca ^? Ko*# would you 

do it? (follow upt why?) 



V9)At is tho bast definition for th* 

tttra ^ 7 

(«) 

(b) 
(c) 



What Aro th« 

of 



<c ttle»ants (in9rodiantt) 



U^t is tho Mjor difforanc« b«twa«n 
and 7 



(a) 

(b) 
(c) 



What is th« ttttin point of 7 

Which of thtt foUowing is a liko]y rssult 
of 7 

M 

(b) 



Hsro is ths i'jsusi 

Which sids atre you on tuid vhy7 



Givs a S5>««ch/plAn s d«bat(9 in vhich you 

uss what you know about 

to judga ita • 

GivQ rsasons to avploin your point of 
viow. 
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Study Sf^p 4: Adding Variety t o the Queations 

NOW lets oovQ from the • formula* chart to the generation of charts with a 
greater variety of questions, The first key to expanding the range of 
questions you can pose is to focus on the trigger or action verb used to 
describe the problem to the student. Start with these and add soae of your 
own if you can: 



If you want to 
eeasure; 

Recall 



Use these key words 
in the exercise ;_ 



define 

identify 

label 

list 

name 

repeat 



whj>.t 
when 
who 
iJhere 



Illustration 

List the naaes of 
the sain characters 
in the story* 



Analysis 



subdivide 

breakdown 

^^parate 

categorize 

sort 



classify 
group 



Break the story dow\v 
into different 
parts* 



Cosrpariflon 



CQopare 
contrast 
differentiate 
distinguish 



Compare tnemes of 
these two stories* 



Inference 



deduce 

predict 

infer 

speculate 

anticipate 

what if... 



apply 
conclude 



How might we make 
this character 
eore bel^eva^^e? 



BvaluAtion 



evaluate 

judge 

assess 

appraise 

defend 

decide 



argue 
recoGsaend 
debate 
why 

critique 



Evaluate this 
story. Is it well 
written? Why .or 
why not? 
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STIGGiaS/QUELl^MALZ 
HANDOUT 



Th«j oocond kty to expand irq the range of questions you can pose is to plug 
these action words into a growing list of generic questions. Again, consider 
these and add some of your own if you can: 
Recall 



0 Define the word 
o Ifhat is a 



o Label the following • 

e Identify the in this 

0 Who did ^? 

Analysis 

9 What are the basic elements (ingredients) in a 

® What ia/are the functions of 

o Inventory the parts of • 



0 



Categorize the 



e Sort the 



0 Analyze the following * 
0 Classify • 
CoB^arison 

o Cospara the before and after. 

0 Contrast the to the _____ 



0 Differenti te between __„____^ ^ 
Inference 

® Hypothesise what will happen if ^___„__ 
0 Predict what would be true if 



0 Conclude what the result will be if 



0 



What if had happened instead? 
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STIGGINS/QUELLMALZ 
HANDOUT 



e What do«s this information suggest? 
o Given this situation (problem) what should you do? 
Q What rule applies in this case? 
Evaluation 

e What do ^ m believe about ^ 
m Judge what woidd b® the best way to solve the problem 
of . Why did you decide that? 

o Evaluate whetter you would ■ 

in this situation. Why? 

^ Decide if ^ ^^^^ 

use these lists of action verbs and questions to generate a complete chart on 
another topic of relevance to you. 
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Assessment Strategies to High Order Thinking Skills 
Catherine Ross 



Catherine Ross opened her presentation by asking two key questions educators 
should consider when addressing critical thinking skills: 



- What do we mean by evaluation? 

- What behaviors do we seek to evaluate when we evaluate "higher level 
thinking skills"? 

Experts in the field have many definitions of higher level thinking skills; Ross 
discussed several of them. She prefers the question posed by Dr. Arthur Costa: ^'How do 
students behave when they don't know?" 

Ross then had the audience engage in an activity, which would also be useable in 
the classroom, where they "did not know" the answer, and had to use higher level 
thinking skills to determine it. The group then discussed the processes they had used to 
solve the problem. Ross suggested that these were the "behaviors" we seek to measure 
when we attempt to evaluate higher level thinking skills. She emphasized that classroom 
observation of students' growth in these skills can be as effective in evaluation as can 
the paper/pencil test. Teachers should look for such evidence as 1) increased 
perserverance when working on problems, 2) flexibility in trying different approaches to 
solving problems, 3) enjoyment of the process of working on problems that are not 
immediately solvable and 4) transference of problem-solving skills developed in one 
situation to another, dissimilar situation. 

For the last portion of the presentation, Ross went through the eight sections of 
the Ross Test of Higher Cognitive Processes, giving sample items from each section. 

Analogies 

Missing Premises 

Deductive Reasoning 

Abstract Relations 

Sequential Synthesis 

Questioning Strategies 

Relevant and Irrelevant Information 

Analysis of Attributes 
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Assessing Higher Order Thinking Skills 
Issues and Practices 



October 1 and 2, 1987 



Mcnarch Hotel 
Clackamas, Oregon 



sponsored by 
Northwest Regional Educational Laboratory 
and 

ESD #112, Vancouver, Washington 
Idaho Association of Secondary School Principals 
Northwest Evaluation Association 
Oregon Department of Education 
Washington Office of the Superintendent 
of Public Instruction 
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The Northwest Regional Educational Laboratory, in conjunction with five regional associations and 
depirtments of education, is pleased to welcome you to two days of presentations and workshops on 
assessing higher order thinking skills (HOTS;. Good assessment Is essential if we are going to make 
decisions about the instructional needs and progress of students. One goal of this conference is to 
assist people who are deciding how to assess this critical area in our schools to make more informed 
decisions and to produce higher quaiity ascessments. Another goal of this conference is to bring 
together r ple interested in the assessment aspects of thinking skills. One outcome might be a 
cooperative effort in item banking. 

The conference has been di\ided u:to two parts. Issues and Practices. The afternoon of the first 
da\ is designed to elicit perspecti\es about assessing higher order thinking skills. Issues such as 
how should HOTS be assessed, the qualit\ of existing tests, how do we define HOTS and what will 
the future bring will be discussed. This part w:" -'^t the stage for the second da\'s acti\ities. 

The second da> emphasizes practices. There are three strands: daily, less formal classroom 
assessment, formal assessment procedures that ma\ be used at the district le\el, and training on how 
to write test questions which assess HOTS. 

The presenters and participants in the eighteen sessions will make each of the sessions a worthwhile 
exploration of the issues and practices for assessing higher order thinking skills. 

Once again, welcome! We are looking forward to a productive conference. 




Gary Estes 

Director, Evaluation and Assessment 
Northwest Regional Educational Laboratory 
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Conference Schedule at a Glance 
Thursday, October 1 

12:00 Registration 
1:30 Welcome 

1:45 Keynote Address: Stuart Rankin 
2:45 BREAK 

3:00 Panel Response: Kenneth Bunigarner, Stephen P. Norri?, Edys S. Quellnialz 
4:00 Discussion 
4:45 Social 

Friday, October 2 

8:30 Refreshments 

9:00 Writing Multiple-choice Critical Thinking Items (part 1): Stephen P. Norris 

In-clas. om. Informal Assessment of Students' Thinking Skills: Kenneth Bumgarner 

How to Take Unlikely-Looking Textbook Material and Make it Testable for Critical 
Thinking: Connie Missimer 

How to Select a Higher Order Thinking Skills Test: Judith A. Arter 
10:15 BREAK 

10:30 Writing Muhiple-choice Critical Thinking Items (part 2): Stephen P. Norris 
Diagnosing Thinking Deficiencies: Selma Wassermann 
Assessing Creativity: Anita Halstead 

Assessing HOTS in Hawaii: One State's Experience: Judith A, Arter and Lynae S. Paule 
12:00 LUNCHEON 

1:30 Alternative Item Formats for Assessing HOTS (part 1): Edys S. Quellmalz 
Diagnosing Thinking Deficiencies: Selma Wassermann (repeat) 

Test Publishers' Panel. Academic Th:rapy, Psychological Corporation, Riverside, and SRA 
Item Banking and Higher Order Thinking Skills. Richard \V. Naccarato and Ra\ K. Miller 
2:45 BREAK 

3:00 Alternative Item Formats for Assessing HOTS (part 2): Edys S. Quellmalz 
Assessment Strategies for Higher Order Thinking Skills: Catherine Ross 
Classroom Assessment of Higher Order Thinking Skills: Richard Stiggins 
Designing a Testing Program: Wayne Neuberger 
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MONARCH HOTEL CONFERENCE ROOMS 

u ' — I- , 



WILLAMETTE j 




SANDY 



The 
Pacific 
Ballroom 



COLUMBIA 



Sky Bridge to 
Exhibition Hall 



CLACKAMAS 



ISL 



TOLOBB) 



Foyer 



Tiniberline Room 
Room 318 
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DESCRIPTION OF SESSIONS 
Thursday, October 1 
1:45-3:00 Keynote Address: Stuart Rankin 

Colun.bia Rankin Null identify the major issues thai need attention in assessing student 
Room thinking and vill suggest positions on those issues. He uill consider. 1) content 

(What are the thinking skills?); 2) implications of assessment procedures and results 
for instruction; 3) issessment within or across disciplines; and 4) measurement 
questions. 



3:15-4:30 Panel Response and Discussion: Ken Bumgarner, Stephen P. NorrJs, and 

Columbia Edys Quellmalz. Moderator: Vida Taylor. 

Room 



4:45 Social Hour (no host) 

SandN You are in%ited to join workshop presenters and participants for hors d'oeuNrc and 

Room a no host bar. 



9 




Clackamas 
Room 



0 



^^'illamette 
Room 



Sandy 
Room 



Multnomah 
Room 



CONCURRENT WORKSHOP SESSIONS 

Friday, October 2 
9:30 - 10:15 

Writing Muhiple-Choice Critical Thinking Items (part 1): 
Stephen P. Norris. 

This session will concentrate on writing multiple-choice items for assessing specific 
aspects of critical thinking such as induction, credibility of authorities and 
observation, and assumption identification. Explanations will be given for why 
certain sorts of item formats are not suitable for assessing these aspects of critical 
thinking. More suitable formats will be described and some practice time for 
writing items will be provided. 

In-classroom, Informal Assessment of Students' Thinking Skills: 
Kenneth Bumf,arner. 

This session will focus on the ways classroom teachers can assess whether their 
students' thinking skills are improving, using tools/techniques readily available to the 
classroom teacher, such as interviews, observation of student pair learning, use of 
thinking logs, etc. The assessment emphasis will be on "how students behave when 
they don't know the answer" as Art Costa puts it. Participants will be invited to 
share in-classroom techniques they use to get an idea of whether their students' 
thinking is improving. 

So>v's Ears Into Silk Purses - How to Take Uniikel> -Looking Textbook Material 
and Make it Testable for Critical Thinking: Connie Missimer. 
One of the biggest problems teachers face is the fact that many textbooks do not 
readily lend themselves to critical thinking. This workshop will be "hands-on," 
showing some techniques for pulling critical thinking out of unlikeh places, then 
offering everyone the opportunity to try their hands at some additional examples. 

How to Select a Test of Higher Order Thinking Skills: Judith A. Arter. 
The Test Center staff at NWREL has developed a checklist de ^d to assist those 
interested in selecting a test of Higher Order Thinking Skills. ii,e checklist includes 
considerations of content, reliability, validity and usability. These issues will be 
explored as the\ apply to currently avp.iiabif* instruments and the issues discussed in 
the keynote address. A selection of JlOTS assessment instruments will be available 
for examination by participants. This sessior* would be appropriate for anyone 
currently trying to select a HOTS test for local use. 
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10:30 - 11:45 

Clackamas Writing Multiple-Choice Critical Thinking Items (part 2): Stephen P. Norris. 

Room This session will describe an approach for obtaining information on the Nalidiix cf 

multiple choice critical thinking items and tests. The approach depends upon asking 
samples of students lo think aloud as they work through trial items. De'^isions to 
modify, retain, or discard items are based upon comparisons between the qualit\ or 
the students' reasoning and their choices of answers. Generally speaking, we want 
qualit> re8Soning to be associated with keyed answers and poor quality reasoning to 
be associat d with ankeyed answers. The presentation will illustrate how the 
approach was used in the design of a test of high school students' ability to assess the 
credibility of observations, but different grade levels and different aspects of critical 
thinking could have been chosen. 

N\illamette Diagnosing Thinking Deficiencies in the Classroom: Selma Wassermann. 

Room Most teachers implicitly understand the relationship between thinking and beha\ior. 

They see, in their day-to-day dealings with students that pupils may behave 
'thoughtlessly" or inappropriately. They observe, to their dismay, that some pupils 
persistently respond to their instructions by saying, "I don't get it" or "I don't 
understand what you mean." Teachers are quick to discern such lapses in thinking. 
"If you would think about that, you'd figure it out for yourself!" But urging a 
student to "think for himself cannot correct those deficits in his higher order 
functioning that produce his dysfunctional behavior. 

This N\orkshop will examine eight dysfunctional classroom behavior patterns that gi\e 
eNidence of pupils' impaired higher order functioning. Based upon the work of 
Louis Raths, the workshop demonstrates the use of classroom observation instruments 
to assess pupils' "thinking-related" behavioral patte;ns and suggests how classroom 
materials and instructional strategies may work to reduce such d\sfunctional 
classroom behavior. 



Sand\ Assessing Creativity: Anita Halstea(?. 

Room This session wiil examine numerous informal measurements to assess creati\e abilities 

in children and adults. The emphasis will be on practical applications in the 
classroom. 

Multnomah Assessing Higher Order Thinking Skills in Ha>>aii: One State's Experience: 

Room Judith A. Arter and Lynde S. Paule. 

NWREL is currenth under contract with the Hawaii Department of Education to 
deNelop multiple-choice questions that measuie decision-making and problem sohing 
skills. The presenter will discuss the procedures, pitfalls and special studies 
undertaken during the course of this development. Sharing of similar experiences b> 
participants will be encouraged. This session would be appropriate for those 
currently considering developing their own HOTS test. 



12:00 

Columbia Room 



LUNCHEON 
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1:30 - 2:45 

Clackamas Alternati>e Item Formats for Assessing HOTS (part 1): Edys S. Quellmalz 
Room In this wokshop participants will develop a variety of approaches for e\aluaii.ic 

students' higher order thinking skilb. Following 3n introduction to the rationale for 
the framework which specifies four broad categories of higher order skills. 
1) analysis, 2) comparison, 3) inference, and 4) evaluation; participants will examine 
examples of alternati\e item formats. The workshop will then pro\ide opportunities 
to de\elop and discuss assessment *asks designed to measure achie\ement in each of 
the four categories. The workshop will emphasize open-ended, constructed formats, 
and the use of higher order reasoning within on-going curricula. 

Willamette Diagnosing Thinking Deficiencies in the Classroom: Selma M assermann. 
Room Repeat of morning session. 

Sandy Test PubSishers' Panel: Catherine Ross, Gale H. Roid, Ve.n Dahl, and 

Room Margaret Hill. Moderator: Ken Bumgarner. 

Representatives from Academic Therapy, Psychological Corporation, Riverside and 
SRA will discuss what is currently available for assessing higher order thinking skilb. 
Achievement tests as we 1 as specific tests of higher order thinking (Watson-Glaser 
and the Ross Test of Higher Cognitive Processes) will be presented. A discuss'on 
will follow. 



Multnomah Higher Order Thinking Skills and Item Banking: Richaid Naccarato and 

Room Ray K. Miller. 

With the recent popularity of item banking at the state and local levels, one might 
question how difficult it \\ould be to identify items that measure higher order 
thinking skills in the various subject areas. Is it be desirable to key test items on 
this dimension? Representatives from NWREL anu NWEA will discuss, both 
formalh and informalh , their awareness of what seems to be happening nationally 
on this topic; some a\ailable item banks that are keyed to the HOTS dimensions, 
strategies for creating cross-referencing to these skills, and generating tests ^^ith 
HOTS dimensions in mind. 

3:00 - 4:15 

Clackamas Alternati\e Item Formats for Assessing HOTS (part 2): Ed\s S. Quellmalz. 
Room This is a continuation of the 1:30 session. 

Willamette Assessment Strategies for Higher Order Thinking Skills: Catherine Ross. 

Room This session will look at asses.>ing higher Je\el thinking skills with sample items on. 

identifying analogous and metaphorical relationships, questioning strategies, attribute 
analysis; determining relevancy of data; and formulating/testing hypotheses. 

Sandy Classroom Assessment of Higher Order Thinking Skillr* Richard Stiggins. 

Room The session will be divided into two parts. The first part will focus on the results of 

a ne\\ study of the extent to which teachers in one school district actually measure 
Higher Order Thinking Skills in their classrooms on a day :o day basis. The second 
pan will be a discussion of strategies for teaching teachers to measure HOTS 
effecti\ely and efficiently. In both cases, the focus will be on assessments devised 
by teachers or selected by teachers from available instructional materials. 

Multnomah Designing a Testing Program to Include Higher Order Thinking Skills: 

Room N\'ayne Neuberger. 

This session will focus on the purposerul uses of tests. It will rev iew the various 
purposes of testing, particularly as they apply to higher order thinking skills. It vvill 
also include procedures for reviewing current testing practices, and planning for 
modification of a tesiing program to address the purposes tc be served. The session 
will focus on practical ways to review and modify testing programs. 
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Presenters 



Judith A, After, PhD 

Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland. Oregon 97204 

Dr. Arter has been associated v,hh NWREL for nine years. During that time she has been inxolved 
in several projects including provision of evaluation and testing technical assistance with the 
Chapter 1 Evaluation Technical Assistance Center, a national study of refugee adult ESL programs, 
and se\eral test development projects for school districts, state departments of education and 
businesses. She is currentl> director of the Test Center at NWREL and has been co-author of two 
coiisumsr guides designed to assist persons select assessment Instruments in the areas of HOTS and 
School Climate. Dr. Arter has 3 bachelor's degree in mathematics from the University of 
California, San Diego, and a PhD in special education from the Universii> of Illinois, Urbana- 
Champaign. 

Kenneth Bumgarner, EdD 

Division of Instructional Programs Si Ser^^ices 

Old Capitol Building, FG-11 

Olynipia, WA ^8504 

Dr. Bumgarner is Director of Basic Education and Curriculum for the Office of the Superintendent 
of Public Instruction (Washington State SEA) and an adjunct faculty member ai Seattle Uni\ersit> 
and St. Martin's College (Olympia), where he teaches classes related to the impro\ement of students' 
thinking skills. He has had a primar> role In leading the state's collaborative effort for impro\ing 
student thinking in the classroom. lie earned his EdD at Seattle University. 

\'ern Dahl 

Riverside ^^jblishing Company 
P.O. Box 4273 
Carmel, CA 93921 

Mr. Dahl earned an MA in Ed; national Ps>cholog\ from the University of Minnesota, He was a 
school counselo; and teacher for eight years before entering the test publishing field. Mr. Dahl has 
worked for Harcourt Brace Jo'.ano\ich, Westinghouse, CTB^ McGraw-Hill and recenth was hired as 
an Executive Consultant for Riverside. 

Anita HalsteaJ 
310 Union Avenue 
Snohomish, WA 98290 

M?. Halstead has a bachelor's d^^gree in Political Science from the Uni\ersit> of Washington, Seattle, 
and a Master's degree in Creative Education from Seattle Pacific University. She has served as a 
research analyst for the CIA, a city councilwoman, college administrator and gifted education 
specialist. She originated CicqumI) Netnotk in 1979, a unique monthly newsletter. In addition, she 
has written handbooks for managers and businesses, classroom curriculums, a children's workbook 
and has produced an audiocassette on creativity. 

Margaret Hill 

Science Research Associates, Inc. 
155 Nor^n Wacker Drive 
Chicago, IL 60606 

Ms. Hill is the Manager of Test De\eloprnent for SRA where she has worked in test publishing for 
15 years. She hai worked with the Survey of Basic Skills and with the SRA Item Bank. She earned 
a MS in Mathematics from the University of Arizona. 
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Ray K. Miller 

Northwest Evaluation Association 
30633 11th Ave. S. 
Federal Way, WA 98003 

Mr. Miller is the Executive Secretar> for the NWEA and an educational consultant for research and 
evaluation and educational measurement. He received his MEd 5n School Psychological Services 
from the University of Oregon and is a licensed psychologist. In the past, Mr. Miller was the 
Director of the Assessment and Evaluation Cooperative at ESD 121 in Washington and a part-time 
instructor at Seattle Pacific University and Central Washington University. 

Connie Missimer 
4836 N.E. 40th Street 
Seattle, WA 98105 

Ms. Missimer is author of Good Arguments: an Introduction to Critical Thinking (Prentice-Hall, 
1986). For 8 years she taught high school and has recently begun work with primary grades. She 
has given numerous workshops with high school faculties. Last springy she offered an assessment of 
critical thinking assessment tests to the Educational Testing Service in Berkeley. 

Richard Naccarato, PhD 
Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR 97204 

After having taught junior and senior high school in Seattle, Dr. Naccarato obtained his PhD in 
educational psychology from the University of Washington. Since then he has coordinated 
placement and proficiency testing and taught measurement courses at the University of Iowa, 
worked (or Educational Testing Service and consulted in testing and evaluation in several foreign 
countries. He currently is a research associate with the Test Center at Northwest Regional 
Educational Laboratory in Portland. 

Wayne Neuberger» PhD 
Director, Assessment and Evaluation 
Oregon Department of Education 
700 Pringle Parkway S.E. 
Salem, OR 97310 

Before assuming his current responsibilities with the State Department of Education, Dr. Neuberger 
was the Coordinator of Program Planning and E\aluation for Beaverton Schools. He is also a 
NWEA Board Member. Dr. Neuberger earned his PhD in Educational Research from New Mexico 
University. 

Stephen P. Norris, PhD 
Center for the Study of Reading 
174 Children's Research Center 
51 Gerty Drive 
Champaign, IL 61820 

Dr. Norris is a professor at the Memorial University in New Foundland and the author of the Te.\, 
on Appraising Observations. He is currently a visiting scholar at the University of Illinois-Urbana 
Center for the Study of Reading. He is working on several other tests of higher order thinkmg and 
is coauthor with Robert Ennis of Exaluating Critical Thinking (to be released this summer). Dr. 
Norris earned his PhD from the University of Illinois-Urbana. 




Lynde S, Pau!e, PhD 

Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR 97204 

Dr. Paule has worked in Evaluation and Assessment at KWREL for the past ten years. She 
developed test items for the states of Alaska, Washington and Havvaii and most recent!}, de% eloped 
a Communications Test for Lewiston, Idaho. She has six vears of classroom teacning experience 
and is currently an Instructor in the Management Di\ision at Mar\lhur$t College. She earned her 
PhD from the University of Oregon. 

Edys S, Quellmalz, PhD 
205 Georgia Lane 
Portola, CA 94025 

Dr. Queunijb is a consultant to several education agencies' higher order thinking projects. She is 
an author of -iiany books and articles on the use of thinking skills across the curriculum, int.iuding 
Measuring Thinking Skills in the Classroom which she coauthored with Richard Stiggins. Dr. 
Quellmalz was formerly a Senior Research Associate with the Center for the Stud> of E\aluaiion, 
UCLA. She earned her PhD in Instructional Research and Development from UCLA. 



Stuart Rankin, EdD 
Deputy Superintendent 
Detroit Public Schools 
5057 \Voodw?.rd 
Detroit, MI 48202 

Dr. Rankin is Deput> Superintendent of Detroit Public Schools. He bimgs to this conference a 
perspective whicii combines curriculum, research and school administration. Dr. Rankin is 
coauthoring a book entitled Dimensions m Thinking. He has also been invohed in a nationwide 
panel examining hinkmg skilk Dr. Rankin earned his EdD at Wayne State University. 

Gale H, Roid, PhD 
Assessment Research 
P.O. Box 8900-324 
Salem, OR 97303-0890 

Dr. Roid has a PhD in ps>chometrics, has been a staff member and consultant to test publishers, 
has worked in statewide assessment, is a researcher with numerous publications in cril<?rion- 
referenced testing, and is author of A Technology of Test-item Wiitiiig (Academic Press, 1982). He 
has assisted Psychological Corporation with several of their achievement and aptitude tefts and he 
will be reprccenting them today on the Test Publishers' Panel. 

Catherine Ross 
15820 6th S.W. 
Seattle, WA 98166 

Ms. Ross is a consultant in the fields of curriculum development and higher order th-nking skills. 
She teaches at Seattle Pacific Uni\ersjt> and is currentl> an administrator in the 'ilme School 
District in Seattle. She i" the coauthor of the Ross Test of Higher Cognitive Piocesses and author 
of Cognit >e Challenge Cards. Ms. Ross received her EdM at University of Washington. 

Richard Stiggins, PhD 
Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR i 204 

Dr. Stiggins is Director of the Center for Performance Assessment at NWREL and has de\ eloped 
strategies for classroom assessment of thinking. He is coauthor with Quellmalz and Rubel of 
Mtasunng Thinking Skills in iht Clasuoom. He earned his PhD from Michigan State Univprsitj 
and is a member of the facult> in the Graduate School of Professional Studies at Lewis and Clark 
College. 
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Vida S. Taylor, £dD 
ESD #112 

1313 N.E. 134th Street 
Vancouver. \VA 98685 

Di. Taylor was recenth hired as the Supervisor for Curriculum Services for ESD 112. Dr. Ta\lor 
pre\iously was an Elementar\ Principal for the Redmond, Oregon School District and a part-tme 
instructor at Portland State University. She has an EdD in Educational Administration. 

Selma Wassermann, EdD 
Faculty of Education 
Simon Fraser University 
Burnaby, British Columbia 
V5A 1S6 CANADA 

Dr. Wassermann teaches at Simon Fraser University and has been involved with thinking skills for 
two decades. She and Louis E. Raths coauthored the seminal text, Teaching for Thinking- Theory 
and Application (Charles Merrill, 1966) and after Rath's death. Dr. Wassermann was executive 
author of the extensively revised edition. Teaching for Thinking, Theory, Strategies, and Activities 
for the Classroom (Teachers College Press, 1986). Dr, Wassermann earned her EdD from New York 
University. 
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Sponsors 



Nortluvest Regional Educational Laboratory 

The Northwest Regional Educational Laboratory (NWREL) assists education, government, 
community agencies, business, and labor in improving quality and equality of educational programs 
and processes. NWREL operates the Test Center, a library of assessment instruments and 
information about testing for the Northwestern states and Region IV Chapter 1 programs. NWREL 
is an independent, nonprofit institution governed by a 33-member Board of Directors. 

BSD 112, Vancouver, Washington 

Educational Service District 112 serves 31 public school districts in six counties in southwest 
Washington. It is one of 9 regional ESDs in the state. In addition, the ESD provides services to 
private schools and two state schools, and it functions as a liaison with universities and area 
community and industrial organizations. Approximately 80 percent of all ser\ices pro\ided b\ ESD 
112 are local district cooperatives designed to meet the expressed needs of the district's staff and 
students. 

Idaho Association of School Administrators 

The Association is a professional organization ser\ icing the needs of public school superintendents, 
ele.Tientary and secondary school principals, and special education administrators in the state of 
Idaho. 

Northwest Evaluation Association 

The Northwest Evaluation Association (NWEA) is a consor lum of school districts in the northwest 
devoted to the general improvement of educational assessment and evaluation. In addition, NWEA 
has developed assessment products including test item banks for the basic skill areas of reading, 
language arts, and mathematics. The Science Curriculum and Assessment Project has de\ eloped a 
7,00f 'tern pool. These item banks are Rasch calibrated and have been pilot tested extensively. 

Office of the Superintendent of Public Instruction, Washington 

The OSPI provides leadership in the state of Washington for the enhancement of thinking skills 
instruction. Two statewide conferences as well as numerous workshops ha\e added impetu*= io this 
statewide thrust. Thinking skills objecti\es are integra.ed into all content areas in state curriculum 
guides. 

Oregon Department of Education 

The state of Oregon places a hea\\ emphasis on thinking skills in its curriculum. This is reflected 
both in the way in which test results are anahzed and in the state assessment tests current!) under 
development. 
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DR. JUDITH A. ARTER 
NORTHWEST REGIONAL 
EDUCATIONAL LAB 
101 S.W. MAIN ST. 
PORTLAND, OR 97204 

DR. WILLIAM AUTY 
CORVALLIS, 0F£. SCH. DIST. 5091 
1555 S.W. 35TH STREET 
CORVALLIS, OR 97333 

MR. FRED BANNISTER 

WASH. OFFICE/SUPT. OF PUBLIC 

INST. 

OLD CAPITOL BUILDIVG, FG-U 
OLYMPIA, WA 98504 

MR. ROGER BERG 

ALOHA HIGH SCHOOL, ALOHA, 

ORE. 

P.O. BOX 200; SCH. DIST. #48 
BEAVERTON, OR 97075 

MR. TOM BESSON 

SCIENCE RESEARCH ASSOCIATES, 

INC. 

2505 MADISON ST. 
EUGENE, OR 9"/ 405 

MR. ERIC BIGLER 
CLACKAMAS BSD 
P.O. BOX 216 

MARYLHURST, OR 97036 

DR. DIAN BLOM 

KENT, WASH. PUBLIC SCHOOLS 

12033 S.E. 256TH 

KENT, WA 98031 

MR. CHARLES BLONDINO 
SEATTLE, WASH. ESD NO. 121 
1410 S. 200TH STREET 
SEATTLE, WA 98198 

MS. SANDY BLONDINO 
SUMNER SCHOOL DISTRICT #320 
1202 WOOD AVENUE 
SUMNER, WA 98390 

MR. PAUL M. BOLY 
SUNSET HIGH SCHOOL 
RTE. 2 BOX 612 
HILLSBORO, OR 97123 



MR. RICHARD BRANNAN 
PORTLAND PUBLIC SCHOOLS 
3830 S.E. 14TH WENUE 
PORTLAND, OR 97202 

DR. KENNETH BUMGAR'NER 
BETHEL SCHOOL DISTRICT #403 
516 EAST 176TH 
SPANAWAY, WA 98387 

MS. SUE CHADWICK 
PASCO, WASH. ESD. NO. 123 
124 S. 4TH AVENUE 
PASCO, WA 99301 

MS. NANCY CONGLETON 
SUMNER .<^CHOOL DISTRICT #320 
1202 WOOP AVENUE 
SUMNER, WA 98390 

MR. GEORGE CRISP 
CLACKAMAS, ORE. HiGH SCHOOL 
13ft01 S. E ""^PSTER ROAD 
MlLWAUKi^. OPv 97267 

MR. NELS DOELEMAN 
ALOHA HIGH SCHOOF , ALOHA, 
ORE. 

P.O. BOX 200; SCH. DIST. #48 
BEAVERTON, OR 97075 

MS. ELLEN V.DDY 

SPANAWAY, WASH. BETHEL SCH. 

DIST. 

516 E. 176TH STREET 
SPANAWAY, WA 98387 

DR. GARY ESTES 
NORTHWEST REGIONAL 
EDUCATIONAL LAB 
101 S.W. MAIN STREET 
PORTLAND, OR 97204 

MS. DOROTHY F/ NNING 
KENNEWICK, WALH. SCHOOL 
DISTRICT 

10205 MAPLE LANE 
PASCO, WA 99301 
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MR. GLEN FIELDING 
OREGON STATE BOARD OF 
EDUCATION 

345 NORTH MONMOUTH AVENUE 
MONMOUTH, OR 97361 

MR. LANDON A. FITCH 

WENATCHEE, WASH. SCHOOL 

DISTRICT 

1101 MILLERDALE 

WENTACHEE, WA 98801 

MS. JILL FOLTZ 
GRANTS PASS, ORE 
N. VALLEY H. S. 
6741 MONUMENT DRIVE 
GRANTS PASS, OR 97526 

MR. DAVID GODFREY 
MILWAUKTE, ORE. HIGH SCHOOL 
11300 S. E. 23RD • 
MILWAUKIE, OR 97222 

MS. MARCIA HAACK 
BEAVERTON, ORE. SCHOOL 
DISTRICT 
P.O. BOX 200 
BEAVERTON, OR 97075 

MR. MARK HADDOCK 
OL\MPIA, WASH. PUBLIC SCHOOLS 
1113 E. LEGION WAY 
OLYMPIA, WA 98501 

MS. ARCELLA HALL 

PORT ANGELES, WASH. HIGH 

SCHOOL 

304 E. PARK AVENUE 
PORT ANGELES, WA 98362 

MS. ANITA HALSTEAD 
CREATIVITY NETWORK 
310 UNION AVENU£ 
SNOHOMISH, WA 98290 

MS. MARILYN S. HARTZELL 
NORTHWEST REGIONAL 
EDUCATIONAL LAB. 
101 SW MAIN, SUITE 500 
PORTLAND, OR 97204 • 



DR. WALTER HAfHAWAY 
PORTLAND PUBLIC SCHOOLS 
501 N. DIXON STRF.ET 
PORTLAND, OR 97227 

MS. BE"*^ HFNDERSON 
PASCO, WASHINGTON ESD 123 
124 S. 4TH 
PASCO, WA 99301 

DR. MARSHALL HERRON 
CORVALLIS, ORE. SCH. DIST. 509J 
1555 S.W. 35TH STRE^^T 
CORVALLIS, OR 97333 

MS. MARGARET HILL 

SCIENCE RESEARCH ASSOCIATES, 

INC. 

155 NORTH WACKER DR. 
CHICAGO IL 60606 

DR. CAROLYN S. HUGHES 
OKLAHOMA CITY PUBLIC SCHOOLS 
900 N. KLEIN 

OKLAHOMA CITY, OK 73106 

MS. LINDA HUGLE 
JOSEPHINE CTY. ORE. SCHOOL 
DISTRICT 
706 N.W. A 

GRANTS PASS, OR 97526 

MS. MELIKJA IRWIN 
CLACKAMAS, ORE. HIGH SCHOOL 
13801 S. E. WEBSTER ROAD 
MILWAUKIE, OR 97267 

DR. MARY M. JOHNSON 
CORVALLIS, ORE. SCHOOL DIST. 
509J 

1555 SW 35TH 
CORVALLIS, OR 97330 

MS. PEGGY JOHNSON 

LAKE STEVENS, WASH. SCHOOL 

DISTRICT 

12708 20TH STREET N.E. 
LAKE STEVENS, WA 98258 

MS. BARBARA JOHNSTONE 
WASH. OFF. OF SUPT. OF PUBLIC 
INST. 

OLD CAPITOL BUILDING, FG-11 
OLYMPIA, WA 98504 
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MS. PAM KAISER 

LAKE STEVENS, WASH. SCHOOL 

DISTRICT 

12708 20TH STREET N.E. 
LAKE STEVENS, WA 98258 

MS. JUDY KING 
SUMNER SCHOOL DISTRICT #320 
1202 WOOD AVENUE 
SUMNER, WA 98390 

DR. GAGE KINGSBURY 
PORTLAND, ORE. PUBLIC SCHOOLS 
501 N. DIXON SE 
PORTLAND, OR 97227 

MS. JEAN KIRSCH 

PORTLAND, ORE. MULTNOMAH 

CTY. ESD 

220 S. E. 102ND 

PORTLAND, OR 97216 

MR. ROB LARSON 

OREGON MUSEUM OF SCIENCE & 

INDUSTRY 

4015 S. W. CANYON 

PORTLAND, OR 97221 

DR. RICHARD LAUGKLIN 
CLATSOP CTY. ORE. E.D SUPT. 
3199 MARINE DRIVE 
ASTORIA, OR 97103 

MS. JANET L LAUGHLIN 

ASTORIA, ORE. EDUCATIONAL 

SERVICES 

RTE. 1, BOX 91 3E 

ASTORIA, OR 97103 

MR. JAY LEET 

TIGARD SCHO '^L DIST.R1CT 23J 
13137 S. W. PACIFIC HI'/HWAY 
TIGARD, OR 97223 

DR. THOMAS S. LINDERSMITH 
LAKE OSWEGO SCHOOL DIST. 7J 
P.O. BOX 70 

LAKE OSWEGO, OR 97034 
MR. TOM LOWE 

WASHOUGAL, WASH. HIGH SCHOOL 
1201 39TH STREET 
WASHOUGAL, WA 98671 



MR. JOHN MADISON 
BEAVERTON, ORE. HIGH SCHOOL 
P.O. BOX 200 
BEAVERTON, OR 97223 

MS. E. ANN MARCHBANK 
TYEE, WASH. HIGH SCHOOL 
4424 SOUTH 188TH 
SEATTLE, WA 98188 

MS. ELAINE MATTSON 
BEAVERTON, ORE. SCHOOL 
DISTRICT 
P.O. BOX 200 
BEAVERTON, OK 97075 



MS. SUZY MCCAUSy AND 
NORTHWEST REGIONAL 
EDUCATIONAL LAB 
101 S.W. MAIN STREET 
PORTLAND, OR 9720^^ 

MS. DONNA MCDONOUGH 
CLACKAMAS, ORE. COMMUNITY 
COLLEGE 
P. O. BOX 477 
BORING, OR 97009 

MR. JOHN MCMAHAN 
PORTLAND, ORE. PUBLIC SCHOOLS 
5700 N. E. 39TH 
PORTLAND, OR 97211 

MR. RICHARD MEINHARD 
OREGON DEPARTMENT OF 
EDUCATION 

700 PRINGLE PARKWAY SE 
SALEM, OR 97310 

MR. RAY K MILLER 
NORTHWEST EVALUATION 
ASSSOCIATION 
30633 ilTH AVE. S. 
FEDERAL WAY, WA 98003 

MS. CONNIE MISSIMER 
4836 N.E. 40TH STREET 
SEATTLE, WA 98105 
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MS. STEPHANIE MITCHELL 
PORTLAND PUBLIC SCHOOLS 
501 N. DIXON 
PORTLAND, OR 97227 

MS. CAROLYN MOILANEN 
PORTLAND, ORE. PUBLIC SCHOOLS 
501 N. DIXON SE 
PORTLAND, OR 97227 

MS. JANE MONROE 
CLACKAMAS, ORE. HIGH SCHOOL 
13801 S. E. WEBSTER ROAD 
MILWAUKIE, OR 97267 

MR. DAVID G. MOORE 
ALBANY, ORE. LiNN-BENTON ESD 
905 4TH AVENUE SE 
ALBANY, OR 97321 

DR. RICHARD NACCARATO 
NORTHWEST REGIONAL 
EDUCATIONAL LAB 
101 S.W. MAIN ST. 
PORTLAND, OR 97204 

MR. TOM NEAL 

THE DALLES, ORE. WILSON ELEM. 
SCH. 

1413 W. 12TH 

THE DALLES, OR P"058 

MS. KAREN NELSON 
BEAVERTON SCHOOL DISTRICT 
P.O. BOX ZOO 
BEAVERTON, OR 97075 

DR. WAYNE NEUBERGER 

OREGON DEPARTMENT OF 

EDUCATION 

700 PRINGLE DRIVE 

SALEM, OR 97310 

MR. FRED E. NEWTOK 
PORTLAND, ORE. MULTNOMAH 
CTY. ESD 

220 SE 102ND STREET 
PORTLAND, OR 97216 

MS. SHARON NORLEN 
CLACKAMAS, ORE. F'3H SCHOOL 
13801 S. E. WEBSTER ROAD 
MILWAUKIE, OR 97267 
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DR. STEPHEN P. NORRIS 

CENTER FOR THE STUDY OF 

READING 

51 GERTY DRIVE 

CHAMPAIGN, IL 61820 

MS. DORIS O'HARA 
CLACKAMAS, ORE. HIGH SCHOOL 
I380j S. E. WEBSTER ROAD 
MiLWAUKj'E, OR 97267 

MS. DOROTHY PATRICK 
PORTLAND, ORE. CENTENNIAL 
SCH. DIST 

18135 S. E. BROOKLYN STREET 
PORTLAND, OR 97236 

DP LYNDE S. PAULE 
NORTH',, £ST REGIONAL 
EDUCATIONAL LAB 
101 S.W. MAIN ST. 
PORTLAND, OR 97204 

MR. LEON PAULSON 
MULTNOMAH CTY., ORE. ESD 
220 S. E. 102ND 
PORTLAND, OR 97216 

MS. SANDRA PELLENS 
OREGON DEPARTMENT OF 
EDUCATION 

700 PRINGLE PARKWAY S.E. 
SALEM, OR 97370 

MS. EILEEN PETERSEN 

LAKE STEVENS, WASH. SCHOOL 

DISTRICT 

12708 20TH N.E. 

LAKE STEVENS, WA 98258 

DR. EDYS S. QUELLMALZ 
205 GEORGIA LANE 
PORTOLA, CA 94025 

DR. STUART RANKIN 
DETROIT PUBLIC SCHOOLS 
5057 WOODWARD 
DETROIT, MI 48202 

MR. GORDON T. REEo 
SANDY, UT HIGHLAND HIGH 
SCHOOL 

1143 ' MT. RIDGE CIRCLE 
SANDY, UT 84092 
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DR. GALE H. ROID 
P.O. BOX 8900-324 
SALEM, OR 97303 



MS. CATHERINE ROSS 
15820 6TH S.W. 
SEATTLE, WA 98166 



MS. JENNFER R. SALMON 
NORTHWEST REGIONAL 
EDUCATIONAL LAB 
101 S.W. MAIN ST. 
PORTLAND, OR 97304 

MR. STEPHEN SLATER 
OREGON DEPARTMENT OF 
EDUCATION 

700 PRINGLE PARKWAY SE 
SALEM, OR 97310 

MS. MARGE SMITH 
BEAVER'. JN, ORE. SCHOOL 
DISTRICT 
P. O. BOX 200 
BEAVERTON, OR 97075 

MS. MARLENE SODERQUIST 
BEAVERTON, ORE. SCHOOL 
DISTRICT 
P.O. BOX 200 
BEAVERTON, OR 97075 



MS. JILL SORENSEN 

SANDY, UTAH HIGHLAND HIGH 

SCHOOL 

709 DOUGLAS STREET 
SALT LAKE CITY, UT 84102 

DR. RICHARD STIGGINS 
N0RTHWa.3T REGION 
EDUCATIONAL LAB 
101 S.W. MAIN ST. 
PORTLAND, OR 97204 

MR. TOM STRAUGH 
ANCHORAGE, AK. SCHOOL 
DISTRICT 

4600 DEBARR, P.O.BOX 1966.4 
ANCHORAGE, aK 99519 



Mi>. SUE SWIIA 
KENNEWICK, WASH. SCHOOL 
DISTRICT 

1208 W. 19TH AVENUE 
KENNEWICK, WA 99336 

MS. SUE TARRANT-BERG 
ALOHA HIGH SCHOOL 
P.O. BOX 200; SCH. DIST. #48 
BEAVERTON, OR 97075 

DR. VIDA S. TAYLOR 
ESD. NO. 112 
131? N.E. 134Tri STREET 
VANCOUVER, WA 98685 

MR. JIM THOMAS 

LAKE OSWEGO, ORE. SCHOOL 

DISTRICT 

18477 S.W. TIMBERGROVE COURT 
LAKE OSWEGO, OR 97035 

MS. JAN TIETZ 
HIGHLINE PUBLIC SCHOOLS 
15675 AMBAUM BLVD., SW 
SEATTLE, WA 98166 

MS. NANCY VAN KANNEL 
TIGARD, OREGON HIGH SCHOOL 
9000 JS.W. DURHAM ROAD 
TIGARD, OR 97226 

MS. DIANE WALK£R 
BETHEL SCK. DIST. 
516 E. 176TM STREET 
SPANA\/A^ , WA 98387 

DR. JULIA C. WAN 
BAINBRIDGE ISL. SCHOOL DIST. 
8489 MADISON AVENUE NE 
BAINBRIDGE ISLAND, WA 98110 

DR. SELMA WASSERMANN 
SIMON ERASER UNIVERSITY 
BURNABY, BC V5A 1S6 

MS. ANNE S. WAX 
TIGARD SCHOOL DISTRICT 23J 
13137 S. W. PACIFIC HIGHWAY 
TIGARD, OR 97223 

MS. RUTH WEHLING 
1012 N. JANTZEN 
PORTLAND, OR 97217 
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MR, KEN WELLS 
CLACKAMAS, ORE. HIGH SCHOOL 
!380i S. E. WEBSTER ROAD 
MILWAUKIE, OR 97267 

MR. E. PAUL WILLIAMS 
PORTLAND, ORE. COMMUNITY 
COLLEGE 

12000 S. W. 49TH AVENUE 
PORTLAND, OR 97219 

DR. BARBARA WOLFE 
OREGON DEPARTMENT OF 
EDUCATION 

700 PRINGLE PARKWAY SE 
SALEM, OR 97310 

MR. PETER WOLMUT 
PORTLAND ORE., MULTNOMAH 
CTY. ESD 
P.O. BOX 16657 
PORTLAND, OR 97211 

MS. ILA ZBARASCHUK 
AUBURN, WASH. ADVENTIST 
ACADEMY 

5000 AUBURN WAY SOUTH 
AUBURN, WA 98002 

MR. ROGER ZIEGELMAN 
PSYCHOLOGICAL CORPORATION 
2114-7TH AVE. W. 
SEATTLE, WA 98119 
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